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Executive  Summary 


ADVANCED  MESOSCALE  WEATHER  FORECASTING  TO  SUPPORT 
TACTICAL  OPERATIONS  ON  THE  AIRLAND  BAT  iLEFIELD 


The  wide  dispersion  of  combat  forces  on  the  modem  battlefield  and  the  complexity  of 
electro-optical  weapon  systems  have  increased  the  need  for  forecasted  weather 
parameters  that  could  be  used  as  Input  to  Tactical  Decisions  Aids  (TDA).  The 
Integrated  Meteorological  System  (IMETS)  will  enable  the  Air  Force  Staff  Weather 
Officer,  who  is  assigned  to  Army  Divisions  and  Corps,  to  provide  weather  forecasts  and 
weather  effects  Information.  This  Innovative  research,  designed  to  support  IMETS,  used 
three  advanced  software  technologies  (i.e.,  a  traditional  neural  net,  a  hybrid  neural  net, 
and  image  understanding)  to  provide  rapid  and  accurate  tactical  weather  forecasting 
models.  CCI  believes  they  are  foundation  technologies  for  advanced  weather  forecasting 
and  satellite  image  processing. 

A  traditional  neural  net  leaning  algorithm,  back-propagation,  was  used  to  create  an 
optimal  weather  forecasting  model  (BP-Atlanta)  whose  performance  could  be  directly 
compared  to  the  GOPAD-Atlanta-RIR  model  developed  during  Phase  I,  the  LFM/MOS, 
and  the  National  Weather  Service  Forecast  Office  (NWSFO).  This  investigation  also 
sought  to  observe  how  different  back-propagation  parameters  affected  the  performance 
of  the  model,  and  whether  the  weighting  values  on  the  Interconnections  between  the 
nodes  might  be  useful  for  identifying  Important  variables. 

The  hybrid  neural  net  approach  Is  based  upon  an  algorithm  called  Goal  Oriented 
Pattern  Detection  (GOPAD),  which  is  technically  described  as  a  statistical,  optimizing, 
machine-learning,  analogue,  forecast  model  creation,  software  tool.  The  output  from 
GOPAD  is  a  tactical  or  mesoscale,  real-time  weather  forecasting  software  program  that 
executes  in  seconds  on  any  computer  and  in  any  language.  During  the  summer  of 
1989,  a  GOPAD  forecast  model  was  independently  tested  by  NOAA  during  a  real-time 
severe  and  significant  weather  forecasting  exercise  called  SHOOTOUT-89  that  took 
place  around  Boulder.  Colorado. 

An  innovative,  non-digital,  symbolic  Image  representation  scheme,  technically  described 
as  a  width-encoded  medial  axis  (WEMA),  was  used  to  perform  Image  understanding  by 
perceptual  grouping.  A  software  program  called  Cloud  Image  Representation, 
Recognition,  and  Understanding  Software  (CIRRUS-I)  was  developed  to  demonstrate 
a  capability  to  Intelligently  track  clouds  in  multi-temporal  satellite  Imagery.  The 
symbolic  representation  of  segmented,  or  sliced,  cloud  regions  makes  it  possible  to 
manipulate  individual  cloud  regions,  to  attach  knowledge  to  cloud  objects,  to  use  shape 
Information,  to  use  the  relative  proximity  of  one  object  to  another,  to  Infer  cloud 
behavior,  to  compute  the  orientation  and  magnitude  of  cloud  displacements,  and  to 
Intelligently  deduce  the  direction  of  movement  of  tracked  clouds.  CIRRUS-I  produces 
cloud/temperature  displacement  vectors  that  could  be  passed  Into  a  vorticity  model  to 
compute  synoptic  wind  patterns  across  the  continent.  These  wind  patterns  could  then 
be  used  as  Input  into  mesoscale  weather  forecast  models. 
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PHASE  n  KNAL  REPORT 


1.0  RESEARCH  OVERVIEW 

The  wide  dispersion  of  combat  forces  on  the  modem  battlefield  and  the  complexity  of 
electro-optical  weapon  systems  have  Increased  the  need  for  forecasted  weather 
parameters  that  could  be  used  as  Input  to  Tactical  Decisions  Aids  (TDA).  The 
Integrated  Meteorological  System  (IMETS)  will  enable  the  Air  Force  Staff  Weather 
Officer,  who  Is  assigned  to  Army  Divisions  and  Corps,  to  provide  weather  forecasts  and 
weather  effects  Information.  CCI's  research  was  designed  to  support  IMETS  using  two 
advanced  software  technologies  (te.,  neural  nets  and  symbolic  Image  processing)  to 
provide  rapid  and  accurate  tactical  weather  forecasts.  CCI  believes  these  are 
foundation  technologies  for  weather  forecasting  and  satellite  Image  processing. 

1.1  Phase  I  Background 

In  the  fall  of  1987,  the  Atmospheric  Effects  Division,  U.S.  Army  Atmospheric  Sciences 
Laboratory  (ASL),  supported  as  Small  Business  Innovative  Research  (SBIR)  a  Phase  I 
project  by  Consultant's  Choice,  Inc.  (CCI)  to  develop  analysis  and  graphics  techniques 
to  display  current  weather-related  Information  over  a  limited  tactical  area,  and  to 
experiment  with  an  Innovative  proprietary  software  program,  called  Goal  Oriented 
Pattern  Detection  (GOPAD),  for  creating  weather  forecasting  models.  Two  prototype 
software  demonstrations  were  developed  and  delivered  to  the  Government:  (1)  an  MS- 
DOS-based,  attributed  vector  map  representation  that  could  support  an  object-oriented 
programming  paradigm,  and  (2)  a  Probability  of  Precipitation  (PoP)  forecasting  model  for 
Hartsfleld  International  Airport  in  Atlanta,  Georgia. 

The  attributed  vector  map  representation  was  designed  to  demonstrate  a  capability  to 
support  an  advanced  programming  paradigm  that  would  be  created  In  Phase  n  called 
Frame-Based  Reasoning  for  an  MS-DOS  computer.  However,  after  the  Phase  II  was 
awarded,  it  was  determined  that  the  research  should  focus  on  symbolic  Image 
processing  research  that  would  be  more  beneficial  to  the  Government. 

The  GOPAD  -Atlanta-RIR  probability  of  precipitation  (PoP)  model  was  compared  to  the 
forecasts  issued  by  the  National  Weather  Service  Forecasters  and  LFM/MOS  model  for 
June.  July,  and  August  of  1987.  It  is  Important  to  note  that  the  GOPAD  model  used 


data  from  only  three  rawin sondes  from  the  12Z  sounding  to  produce  a  forecast  at 
1230Z  for  the  12-24Z  period,  whereas  the  LFM/MOS  used  the  OZ  sounding  In  order  to 
Issue  a  forecast  at  approximately  the  same  time  as  the  GOPAD  model.  The  performance 
or  skill  of  all  forecasts  models  was  evaluated  based  upon  the  Brier  scores.  The  skill 
scores  for  the  summer.  1987,  were  computed  as  follows:  (a)  LFM/MOS  16.3%:  (b) 
NWSFO  20.9%;  and  (c)  GOPAD  27.6%.  These  preliminary  performance  results  and  the 
potential  to  further  improve  GOPAD  forecast  models  by  adding  new  sources  of  predictor 
variables  were  sufficient  to  justify  further  experimentation  In  Phase  n  with  the  GOPAD 
approach,  and  a  more  traditional  neural  net  approach,  based  on  the  back-propagation 
learning  algorithm.1 


In  the  fall  of  1988,  the  Atmospheric  Effects  Division,  U.S.  Army  Atmospheric  Sciences 
Laboratory  (ASL),  supported  a  Small  Business  Innovative  Research  (SBIR)  Phase  n 
project  by  Consultant’s  Choice,  Inc.  (CCI)  to  Investigate  Goal  Oriented  Pattern  Detection 
(GOPAD),  Artificial  Neural  Nets  (ANS),  and  a  symbolic  satellite  Image  processing 
technique  to  support  tactical  weather  forecasting  on  the  AlrLand  Battlefield.  The  overall 
goals  of  the  research  were  to  develop  technologies  that  could  be  used  to  automate  the 
production  of  weather  forecasting  models,  to  evaluate  the  performance  of  prototype 
forecasting  models,  and  to  demonstrate  how  an  Innovative  symbolic  image 
representation  scheme  could  be  used  to  automatically  track  clouds  In  multi-temporal 
GOES  imagery. 

CCI's  neural  net  approach  was  based  upon  an  algorithm  called  Gcal  Oriented  Pattern 
Detection  (GOPAD)  and  the  back-propagation  learning  algorithm.  GOPAD  Is 
technically  described  as  a  statistical,  optimizing,  machine-learning,  analogue,  forecast 
model  creation,  software  tool.  The  output  from  GOPAD  Is  a  tactical  or  mesoscale,  real- 
time  weather  forecasting  software  program  data  that  executes  in  seconds  on  arty 
computer  and  In  any  language.  During  the  summer  of  1989,  a  GOPAD  forecast  model 
was  Independently  tested  by  NOAA  during  a  real-time  severe  and  significant  weather 

uA.hiv.iot  <«ctnuu  ouwxuoi-uo  UAai  uajiw  yiavz  m  uic  iduuiuci  <uiu  j-zciivci, 

Colorado,  area.  CCI  believes  that,  the  GOPAD  technology  will  represent  a  significant 
Improvement  In  the  ability  to  forecast  a  wide  variety  of  mesoscale  weather  phenomena. 
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zine,  April,  1987,  pages  4-22. 
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CCI's  symbolic  Image  processing  approach  was  based  upon  the  Digital  [image!  to 
Symbolic  Image  Transformation  Algorithm  (DSTA).  This  transformation  algorithm 
operates  on  a  segmented  digital  image  to  produce  a  discrete  width-encoded  medial  axis 
(WEMA)  in  a  LISP  list  format.  This  type  of  symbolic  representation  facilitates  image 
understanding  by  perceptual  grouping.2  A  software  program  called  Cloud  Image 
Representation,  Recognition,  and  Understanding  Software  (CERRUS-I)  was 
developed  to  exploit  the  WEMA  by  demonstrating  a  capability  to  Intelligently  track 
clouds  in  multi-temporal  satellite  Imagery.  CIRRUS-I  produces  cloud/temperature 
displacement  vectors  that  could  be  passed  into  a  vortlcity  model  to  compute  synoptic 
wind  patterns  across  the  continent. 

2.0  PHASE  II  RESEARCH  OBJECTIVES 

There  were  five  broad  research  objectives  in  Phase  n. 

a.  To  develop  a  Probability  of  Precipitation  (PoP)  forecast  model,  called  BP- 
Atlanta,  for  Hartsfleld  International  Airport  using  the  back-propagation 
learning  algorithm  (BP)  and  the  same  rawinsonde  data  bases  used  to  train 
and  test  the  GOPAD -Atlanta  model. 

b.  To  compare  the  performance  of  the  BP-Ailanta  forecasts  to  the  performance 
of  GOPAD-Atlanta,  LFM/MOS,  and  the  NWSFO  forecasts. 

c.  To  develop  a  severe  and  significant  mesoscal;  weather  forecasting  model, 
called  GOPAD-RT89,  for  an  NOAA/FSL  sponsored  forecasting  exercise 
called  SHOOTOUT-89. 

d.  To  compare  and  evaluate  the  performance  of  the  GOPAD-RT89  model  to  the 
other  forecast  models  that  participated  in  SHOOTOUT-89. 

e.  To  demonstrate  the  capability  of  a  width-encoded  medial  axis  (WEMA)  to 
track  homogenous  temperature  regions  (i.e.,  clouds)  in  GOES  IR  imagery. 


2Biedennan,  Irving,  Human  Image  Understanding;  Recent  Research  and  a  Theory.  Computer  Vision,  Graphics,  and 
Image  Processing  32, 1985,  pages  329-73. 
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3.0  ARTIFICIAL  NEURAL  NETS 


3.1  Reseaicn  Overview 

Tills  research  was  structured  to  develop,  modify,  test,  evaluate,  and  compare  the 
usefulness  of  Artificial  Neural  Neta  CANS)  as  an  approach  to  developing  tactical 
weather  forecasting  models.  In  Phase  I,  Goal  Oriented  Pattern  Detection  (GOPAD1) 
was  used  to  develop  a  Probability  of  Precipitation  (PoP)  forecast  model  for  Hartsfleld 
International  Airport  In  Atlanta,  Georgia.  The  high  skill  scores  achieved  by  this  hybrid 
neural  net  approach  suggested  that  perhaps  a  more  traditional  neural  net  approach 
using  the  back-propagation  (BP)  learning  algorithm  might  be  able  to  outperform  a 
GOPAD1  model.  In  addition,  GO  PAD  1  (i.e.,  the  BASIC  language  version)  appeared  to 
have  several  problems  that  might  limit  its  usefulness  as  a  tool  for  creating  tactical 
weather  forecasting  models.  Although  many  of  these  limitations  were  eventually 
mitigated  by  the  evolution  of  GO  PAD  1  into  GOPAD2,  we  believed  that  it  would  be 
prudent  to  Investigate  the  potential  of  a  traditional  neural  net  approach,  since  the  data 
bases  were  readily  available  for  direct  comparisons. 

3.2  Training  and  Testing  Data  Base 

The  training  data  base  consisted  of  rawinsor  Je  data  from  three  sites:  Athens,  Georgia; 
Waycross,  Georgia;  and  Centerville,  Alabama,  for  a  7-month  period  (i.e.,  April  to 
October)  and  for  a  10-year  period  (i.e.,  1975  to  1984). 

The  testing  data  base  consisted  of  rawinsonde  data  for  a  5-month  period  (Le.,  May  to 
September)  during  1987.  The  testing  dala  base  consisted  of  the  same  variables  that 
were  In  the  training  data  base. 


Both  data  bases  were  composed  of  the  twice  dally  (ie„  OZ  and  12Z)  rawinsonde 
soundings.  Each  sounding  was  composed  of  four  variables  (i.e.,  temperature,  relative 
humidity,  and  wind  direction  and  speed)  for  ten  levels  in  the  atmosphere  (i.e.,  900, 800, 
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converted  to  u  (east-west)  and  v  (north-south)  components.  All  variables  were 
normalized  to  have  a  mean  of  zero  and  a  standard  deviation  of  one.  Thus,  the  historical 
training  data  base  consisted  of  2,130  records  or  data  points  (i.e.,  213  days /year  x  10 
years),  whereas  the  testing  data  base  consisted  of  153  records.  Each  record  in  the 
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training  and  testing  data  bases  had  241  predictor  variables  (Le.,  4  variables/level  x  10 
levels/rawinsonde  x  3  sites  x  2  rawinsondes/sites)  plus  the  day-number). 

3.3  Ground  Truth  Data  Base 

The  ground  truth  data,  obtained  from  the  National  Climatic  Data  Center  (NCDC), 
consisted  of  hourly  precipitation  data  for  the  National  Weather  Service  Forecast  Office 
at  Hartsfield  International  Airport  in  Atlanta.  Georgia.  The  time  variable  was  shifted  to 
Greenwich  (Z)  time  and  summarized  in  six-hour  totals  ending  at  6Z,  12Z,  18Z,  and  24Z. 
All  variables  were  normalized  to  have  a  mean  of  zero  and  a  standard  deviation  of  one. 

3.4  LFM/MOS  and  NWSFO  Forecasts 

The  forecasts  issued  by  the  National  Weather  Service  Forecast  Office  (NWSFO)  at 
Hartsfield  International  Airport  in  Atlanta  and  the  Limited  Fine  Mesh/Model  Output 
Statistics  (LFM/MOS)  data  were  obtained  from  the  hand-written  reports  maintained  by 
the  NWSFO.  This  information  was  obtained  for  the  3-month  period  June  to  August, 
1987.  'this  data  was  used  as  a  benchmark  during  Phase  I  and  Phase  Q  to  compare  the 
performance  of  GOPAD  and  BP  forecast  models. 

It  is  important  to  note  that  the  LFM/MOS  forecasts  used  the  OZ  sounding  for  the  12- 
24Z  forecast  period,  whereas  GOPAD-ATLANTA  and  BP-ATLANTA  models  used  the  12Z 
sounding  to  produce  the  12-24Z  forecasts.  These  times  were  used  because,  in  an 
operational  setting,  the  LFM/MOS  12-24Z  forecast  based  upon  the  12Z  sounding  is  not 
available  until  about  15-17Z,  whereas  the  GOPAD-ATLANTA  and  the  BP-ATLANTA 
models  are  available  seconds  after  the  12Z  sounding  data  arrives  (l.e.,  about  1230Z). 
Thus,  in  order  for  LFM/MOS  to  provide  timely  Information  to  a  forecaster,  it  must  use 
the  OZ  sounding,  whereas  a  GOPAD  or  BP  model  can  be  based  upon  the  12Z  sounding. 
While  this  difference  may  appear  to  disadvantage  LFM/MOS  vis-a-vis  GOPAD  and  BP,  it 
is  also  important  to  note  the  great  advantage  that  LFM/MOS  has  had  in  research  and 
development  time  compared  ;o  the  limited  experimentation  with  GOPAD  and  BP 
models.  The  point  is  that  perfectly  matched,  objective  comparisons  are  not  yet  possible. 
However,  future  research  and  development  will  eventually  make  direct  quantitative  and 
qualitative  comparisons  possible. 
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3.5  Optimal  Forecast  Model  Research 


3.5.1  Objective 

The  objective  of  this  research  was  to  Identify  the  optimal  combination  of  BP  parameters 
that  would  produce  a  PoP  forecast  model  with  the  highest  possible  Brier-based  skill 
score.3  The  optimal  ANS  model  that  was  produced  is  referred  to  as  BP-ATLANTA-24 1 . 

3.5.2  Research  Results 

This  research  was  performed  during  December.  1988,  and  January,  1989.  The 
approach  used  to  find  the  optimal  parameters  was  trial  and  error.  The  measurement 
used  to  identify  the  optimal  model  was  the  single  (Brier)  skill  score  obtained  for  the 
combined  months  of  May,  June,  July,  August,  and  September,  1987.  The  training  and 
testing  data  bases  were  the  data  bases  that  were  prepared  for  GOPAD  during  Phase  I. 
The  climatology  used  to  compute  skill  score  for  each  of  the  five  months  was  22%.  In 
later  experiments,  the  climatology  was  computed  for  each  month  in  the  training  data 
base.  Consequently,  slight  differences  in  skill  scores  occur  between  the  following  tables 
in  this  section  and  the  tables  in  other  paragraphs. 

The  BP  architecture  and  parameter  settings  for  BP-Atlanta-241  which  produced  the 
highest  Brier-based  skill  score  are  listed  in  Table  1.  The  skill  score  obtained  for  these 
settings  was  20. 19%. 


3Murphy,  A.H.,  and  Daan,  H.,  "Forecast  Evaluation",  Probability.  Statistics,  and  Decision  Making  in  the  Atmospheric 
Sciences.  Westview  Press,  1985,  pp.  379437. 
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TABLE  1 

ARCHITECTURE  AND  PARAMETERS  FOR  BP-ATLANTA-24 1 


3 

Number  of  Input,  hidden,  and  output  layers 

241 

Number  of  nodes  on  Input  layer 

30 

Number  of  nodes  on  the  hidden  layer 

1 

Number  of  nodes  on  the  output  layer 

10 

Number  of  passes 

1 

Number  of  iterations 

0.01 

Learning  rate 

0.90 

Momentum 

Table 

Source  for  Sigmoid  function 

n/a 

Random  seed 

no 

Input  and  output  slabs  connected 

3.5.2. 1  Learning  Rate  Experiment 


The  purpose  of  this  experiment  was  to  observe  the  effects  of  changing  the  learning  rate 
parameter  while  keeping  all  other  network  parameters  constant.  Table  2  shows  Initial 
research  using  the  preprocessed  data  with  fhr  -month  skill  scores,  while  Table  3  shows 
the  weekly  skill  scores  after  the  training  and  testing  data  bas^s  were  reconstructed. 


TABLE  2 

FIVE-MONTH  SKILL  SCORES  FOR  VARIOUS  LEARNING  RATES 

Brier-based 

1  uuPj  1MM  1 

.005 

+  17.79% 

7.11 

.010 

+  20.19% 

7.11 

.020 

+  18.45% 

7.11 

.025 

+  14.07% 

7.11 

.030 

+  13.24% 

7.11 

.040 

+  5.20% 

7.11 

.050 

+  4.06% 

7.11 

.075 

+  2.47% 

7.11 

.100 

-  3.56% 

7.11 

Summer 

1987 

TABLE  3 

WEEKLY  SKILL  SCORES  FOR  VARIOUS 

LEARNING  RATES 

.005  .010  .020  .025  .030 

i  LEARNING  RATES 

.040  .050  .750 

.100 

June  7 

22.1 

18.1 

22.5 

18.6 

15.5 

41.2 

66.2 

74.1 

7.6 

14 

38.7 

32.4 

21.5 

19.3 

22.0 

77.6 

67.1 

66.0 

8.3 

21 

58.9 

62.1 

60.9 

25.6 

38.0 

14.4 

71.5 

74.0 

73.3 

28 

32.8 

40.2 

22.1 

17.4 

34.2 

-22.4 

6.0 

8.7 

53.7 

July  5 

23.4 

30.6 

19.2 

-6.3 

-3.8 

-22.1 

-16.0 

-0.5 

-7.8 

12 

-5.7 

8.9 

13.7 

15.3 

20.3 

-86.7 

-131.0 

-107.5 

-20.9 

19 

27.7 

28.4 

10.6 

4.9 

4.9 

-6.3 

-3.8 

-3.0 

30.0 

26 

-10.2 

-10.2 

-5.5 

-8.0 

-17.1 

15.9 

18.3 

9.3 

-3.6 

August  2 

2.4 

-4.3 

-11.2 

-12.9 

-27.2 

-66.8 

-84.5 

-65.3 

-11.3 

9 

12.0 

17.3 

-2.0 

-3.2 

-5.5 

-21.6 

15.3 

6.1 

1.2 

16 

15.4 

-10.8 

-30.5 

-10.8 

-19.2 

78.5 

49.5 

35.1 

-304.4 

23 

41.0 

50.1 

51.3 

65.4 

75.8 

83.0 

-0.3 

-39.0 

3.1 

30 

-14.0 

-7.5 

2.3 

62.3 

71.2 

91.2 

-99.5 

-217.2 

-102.0 

Averages: 

18.8 

19.6 

13.4 

14.4 

16.1 

13.5 

-3.6 

-12.6 

-21.4 

Combined  Skill  Scores: 
J-J-A  23.9  26.1 

18.6 

9.9 

12.3 

-0.2 

9.5 

8.5 

11.8 

M-J-J-A-S  19.0 

21.4 

19.6 

15.3 

14.5 

6.6 

5.5 

3.9 

-2.0 

3.5.2.2  Hidden  Nodes  Experiment 

The  purpose  of  this  experiment  was  to  observe  the  effects  of  changing  the  number  of 
hidden  nodes  while  keeping  all  other  network  parameters  constant.  Table  4  shows 
initial  research  using  the  preprocessed  data  with  five-month  skill  scores,  while  Table  5 
shows  the  weekly  skill  scores  after  the  training  and  testing  data  bases  were 
reconstructed. 
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TABLE  4 

HIDDEN  NODES  EXPERIMENT 


Hidden  Nodes 

Brier-based 

Skill  Score 

Time  (hr:mln.sec! 

10 

+12.76% 

2.46 

20 

+12.71% 

4.54 

30 

+  20.19% 

7.11 

40 

+18.64% 

9.29 

50 

+14.50% 

11.47 

SO 

+18.44% 

14.04 

70 

+16.52% 

16.22 

80 

+8.87% 

18.41 

93 

+10.37% 

20.58 

100 

+4.84% 

23.16 

TABLE  5 

WEEKLY  SKILL  SCORES  FOR  VARIOUS  NUMBERS  OF  HIDDEN  NODES 

NUMBER  OF  HIDDEN  NODES 

Summer- 


xm 

IQ 

2 Q 

23 

4Q 

5 Q 

SQ 

ZQ 

SQ 

SQ 

June  7 

-1.7 

18.2 

18.1 

28.5 

14.4 

15.7 

6.7 

17.3 

37.4 

14 

15.7 

29.7 

32.4 

18.5 

34.2 

27.3 

26.9 

39.2 

33.1 

21 

65.8 

64.2 

62.1 

56.6 

43.9 

54.4 

54.6 

56.1 

57.6 

28 

51.3 

54.5 

40.2 

44.7 

28.8 

36.9 

39.0 

42.4 

32.9 

July  5 

29.1 

15.6 

30.6 

22.7 

24.2 

22.0 

47.2 

0.2 

18.4 

12 

-19.3 

-19.6 

8.9 

-10.7 

-22.6 

26.1 

6.6 

5.7 

-33.3 

19 

49.5 

33.3 

28.4 

11.6 

7.5 

5.5 

13.4 

15.0 

18.8 

26 

-5.9 

27.7 

-10.2 

18.6 

-3.5 

19.0 

-0.4 

5.1 

0.5 

2 

-3.1 

-12.7 

-4.3 

-11.9 

-31.3 

-34.8 

-53.7 

-35.5 

-37.6 

August  9 

19.4 

5.7 

17.3 

25.6 

26.8 

22.8 

9.7 

5.1 

12.2 

16 

-66.7 

-72.5 

-10.8 

12.9 

1.6 

-63.7 

-83.3 

-127.9 

-28.2 

23 

-2G.7 

30.9 

50.1 

19.9 

-40.0 

40.6 

64.3 

-19.8 

-35.6 

30 

131.2 

23.f 

-7.5 

19.4 

55.5 

-0.2 

34.5 

-70.4 

-86.4 

Averages: 

00 

rH 

t 

15.3 

19.6 

19.7 

10.7 

13.2 

12.7 

-5.233 

l 

o 

bo 

Combined 

Skill  Scores: 

J-J-A 

20.3 

23.8 

26.1 

25.5 

17.5 

23.2 

20.3 

13.8 

15.8 

M-J-J-A-S 

14.C 

14.0 

21.4 

19.8 

15.7 

19.6 

17.7 

10.2 

11.7 

3.5.2.3  Passes  Experiment 


The  purpose  of  this  experiment  was  to  observe  the  effects  of  changing  the  number  of 
passes  in  a  back-propagation  algorithm,  wnile  keeping  all  other  network  parameters 
constant  Table  6  shows  initial  research  using  the  preprocessed  data  with  five-month 
skill  scores,  while  Table  7  shows  the  weekly  skill  scores  after  the  training  and  testing 
data  bases  were  reconstructed. 


Passes 

TABLE  6 

PASSES  EXPERIMENT 

Brier-based 

Skill  Score 

Time  fhnmin.secl 

5 

+19.14 

3.35 

10 

+20.19 

7.12 

20 

+  18.28 

14.22 

30 

+  14.98 

21.34 

40 

+  7.76 

28.46 

60 

+  1.74 

43.08 

TABLE  7 

WEEKLY  SKILL  SCORES  FOR  VARIOUS  NUMBERS  OF  PASSES 
NUMBER  OF  PASSES 


Summer-87 

.5 

JLQ 

2Q 

2Q 

4Q 

m 

June  7 

20.1 

18.1 

30.3 

24.9 

22.7 

17.0 

14 

32.8 

32.4 

27.5 

29.0 

15.0 

27.9 

21 

57.1 

62.1 

55.9 

58.2 

56.3 

52.7 

28 

30.8 

40.2 

26.5 

39.8 

47.5 

40.7 

July  5 

20.2 

30.6 

37.7 

37.7 

33.9 

33.8 

12 

6.1 

8.9 

12.4 

14.7 

27.7 

23.3 

19 

22.3 

28.4 

7.6 

9.0 

6.7 

-4.8 

26 

-9.6 

-10.2 

-19.5 

-20.6 

-24.3 

-27.1 

August  2 

4.3 

-4.3 

-13.5 

-23.9 

-34.8 

-34.1 

9 

10.9 

17.3 

-0.7 

-17.7 

-14.0 

-4.8 

16 

14.5 

-10.8 

-11.8 

-82.9 

-144.7 

-195.6 

23 

53.0 

50.1 

50.1 

19.5 

-102.4 

-178.3 

30 

3.8 

-7.5 

-3.8 

-16.3 

-70.2 

-152.4 

*  n  g 

ICO 

ef  e 

1  O  A 

AA  A 

<c.o.u 

Combined  Skill  Scores: 

lO.U 

1J.U 

'10.9 

OUti? 

J-J-A 

23.8 

26.1 

19.8 

16.6 

10.8 

6.3 

M-J-J-A-S 

20.3 

21.4 

19.5 

6.2 

9.1 

3.2 

3.5.2.4  Momentum  Experiment 


The  purpose  of  this  experiment  was  to  observe  the  effects  of  changing  the  momentum  in 
a  back-propagation  algorithm,  while  keeping  all  other  network  parameters  constant. 
Table  8  shows  initial  research  using  the  preprocessed  data  with  five-month  skill  scores. 


TABLE  8 

MOMENTUM  EXPERIMENT 


Moment  urn 

Brier-based 

Skill  Score 

Ttmg.thraain,§€c) 

.750 

+  16.98 

7.11 

£00 

+  17.96 

7.11 

.850 

+  19.04 

7.11 

.600 

f  20.19 

7.11 

.925 

+  19.79 

7.11 

.950 

+  19.90 

7.11 

3.5.2.5  Iterations  Experiment 


The  purpose  of  this  experiment  was  to  observe  the  effects  of  changing  the  number  of 
iterations  In  a  back-propagation  algorithm,  while  keeping  all  other  network  parameters 
constant  Table  9  shows  initial  research  using  the  preprocessed  data  with  five-month 
skill  scores. 


Iterations 

TABLE  9 

ITERATIONS  EXPERIMENT 

Brier-based 

Skill  Score 

Time  fhrtmin.sec) 

1 

+  20.19 

7.11 

2 

+  14.97 

14.97 

3 

+  9.22 

21.05 

5 

-  8.99 

34.58 
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3.5.2. 6  Sigmoid  Table  Fxperim.mt 


The  purpose  of  this  experiment  was  to  observe  the  effects  of  obtaining  the  Sigmoid  value 
from  a  table,  and  obtaining  the  value  by  calculation  for  a  back-propagation  algorithm, 
while  keeping  all  other  network  parameters  constant.  Table  10  shows  initial  research 
using  the  preprocessed  data  with  five-month  skill  scores. 


TABLE  10 

SIGMOID  TABLE  EXPERIMENT 

Brier-based 

Table 

+  20.19 

Calculated 

+  20.19 

3.5.2.7  Input-Output  Layer  Connection  Experiment 


The  purpose  of  this  experiment  was  to  observe  the  effects  of  not  connecting  the  input 
layer  directly  to  the  output  layer  in  a  back-propagation  algorithm,  while  keeping  all 
other  network  parameters  constant.  Table  11  shows  initial  research  using  the 
preprocessed  data  with  five-month  skill  scores. 


TABLE  11 

LEARNING  RATE  EXPERIMENT 

Brier-based 

Inout-Output 

Not  Connected 

+  20.19 

Connected 

-  29.41 

3.6  Evaluation  and  Comparisons  Research 
3.0.  i  Overall  Gosls 


This  research  sought  to  validate  the  previous  research  work  by  setting  aside  the 
preprocessed  historical  data  base  that  was  originally  created  to  train  the  GOPAD- 


12 


ATLANTA-RIR,  and  reconstructing  a  new  aata  base  from  the  raw  soundings  provided  by 
the  National  Climatic  Data  Center. 

This  research  sought  to  develop  a  combination  of  BP  forecasts  models  that  could  be 
compared  to  the  GOPAD  -E,  -F,  and  -G  models  that  comprise  the  GOPAD-ATLANTA-RIR 
model  developed  during  Phase  I. 

This  research  sought  to  observe  the  degradation  that  would  occur  when  a  12-24  hour 
forecast  model  was  created  rather  than  a  0-12  hour  forecast  model. 

This  research  sought  to  determine  whether  the  weighting  factors  on  the 
Interconnections  In  a  BP  model  could  be  used  to  prioritize  the  relative  contribution  of 
the  input  variables. 

3.6.2  General  Approach 

This  research  was  performed  during  April-May  1989.  A  Hecht-Neilsen  Anza  Plus 
neurocomputer-board  in  an  80286-based  microcomputer  was  used  to  perform  the 
experiments.  The  training  data  base  used  was  reconstructed  from  the  original  raw 
sounding  data  flies. 

Once  the  apparent  optimal  parameter  settings  (listed  in  Table  1)  were  found,  the 
number  of  hidden  nodes,  the  number  of  passes,  and  the  learning  rate  parameters  were 
selectively  varied  to  observe  the  effect  on  weekly  forecasting  skill  scores  for  the  test 
period.  This  task  was  accomplished  by  setting  all  parameters  to  their  optimal  values, 
and  then  varying  the  settings  for  one  parameter  at  a  time.  The  ten-year  historical  data 
base  was  then  processed  through  the  network  to  create  a  forecast  model. 

Once  a  BP  forecast  model  was  created,  the  5-month  testing  data  base  for  1987  was 
processed  through  this  model  one  day  at  a  time  to  produce  the  probability  of  rain  for 
each  day.  The  probability  of  rain  for  each  of  the  153  days  in  the  test  data  base  was 
then  written  to  a  forecast  file.  Each  forecast  file  for  each  of  the  24  models  was  overlaid 
onto  a  standard  LOTUS™  spreadsheet,  where  the  dally,  weekly,  and  monthly  Brier- 
based  skill  scores  were  Immediately  calculated.  The  results  of  these  experiments  are 
shown  in  Tables  3,  5,  and  7. 
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3.6.3  Spreadsheet  Calculation  of  Skill  Score 


Attached  as  Appendix  A  are  samples  of  3  of  the  24  LOTUS™  spreadsheets  that  were 
used  to  calculate  skill  scores  for  the  various  BP  models.  These  spreadsheets 
automatically  calculated  the  performance  of  each  experimental  node!  when  the  BP 
output  file  was  overlaid. 

The  Optimal  Parameters  column  shows  the  optimal  settings  for  BP-Atlanta-241.  The 
column  is  the  same  on  all  spreadsheets. 

The  Variable  Changed  column  shows  which  parameter  was  changed  and  its  setting. 

The  Day  #  column  is  a  day  count  number  for  each  of  the  153  days  or  records  In  the 
testing  data  base. 

The  Forecast  Output  column  is  the  PoP  for  that  day. 

The  Actual  Rain  column  contains  either  a  "I"  or  a  "0"  to  designate  whether  it  rained  or 
did  not  rain  on  that  particular  day  during  the  12Z-24Z  period. 

The  ANS  column  is  the  squared  error  between  the  probability  forecast  column  and  the 
Actual  Rain  column. 

The  Climatology  column  is  the  squared  error  between  climatology  for  each  month  and 
the  Actual  Rain  column.  The  climatology  for  rain  in  May  was  27.7%;  in  June,  it  was 
21.0%;  in  July,  it  was  24.9%;  in  August,  it  was  22.8%;  and  in  September,  it  was  23.0%. 
A  single  climatology  was  used  for  all  months  in  Phase  I  to  compute  skill  scores  for  the 
GOPAD-ATLANTA-RIR  model,  so  the  skill  scores  prepared  in  Phase  n  are  slightly 
different  than  those  reported  in  the  Phase  I  Final  Report. 

The  Daily,  Weekly,  and  Monthly  columns  compute  the  skill  scores  for  the  present  day 
and  the  past  seven  days,  and  for  the  present  day  and  the  past  30  days,  respectively 

The  Brier-based  skill  score  wa.5  computed  according  to  the  following  formula: 

SS  =  1  - 1  (ANS  sq  error/  climatology  sq  error) 
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3.6.4  Comparative  Performance  Evaluations 


3.6.4. 1  Objectives  and  Approaches 

The  objective  of  this  research  was  to  compare  the  performance  of  BP-ATLANTA  models 
to  the  National  Weather  Service  Forecast  Office  (NWSFO),  the  LFM/MOS,  and  the 
GOPAD-ATLANTA-RIR  models  created  during  Phase  I.  The  approach  was  to  create  a 
series  of  BP-based  forecast  models  that  were  similar  in  structure  to  the  GOPAD- 
ATLANTA-RIR  model  developed  In  Phase  I  uting  a  PC/AT  with  a  Hecht-Nellsen 
neurocomputer  board  Installed.  The  performance  of  these  new  models  would  then  be 
compared  to  the  performance  of  the  GOPAD-ATLANTA-RIR,  the  NWSFO.  and  the 
LFM/MOS  forecasts. 

As  an  additional  objective.  CCI  sought  to  verify  the  performance  of  the  BP-AT1ANTA- 
241  model  by  completely  reprocessing  the  original  rawinsonde  training  data  base  ln- 
house.  Consequently,  all  the  preprocessed  historical  data  files  that  were  used  to  find 
the  optimal  parameter  settings  for  BP-Atlanta-241  were  discarded.  In  conjunction  with 
the  reconstruction  of  these  data  bases,  all  forecasts  were  consolidated  Into  LOTUS™ 
spreadsheets  so  that  all  skill  score  computations  would  be  calculated  exactly  the  same. 

3.6.4.2  Research  Results 

Six  BP  forecast  models  were  created  with  the  optimal  parameter  settings  used  by  the 
BP-ATIANTA-24 1  model.  These  models  are  designated  BP-E241,  BP-F241,  BP-G241, 
BP-E32,  BP-F32,  and  BP-G32.  Each  of  these  models  was  trained  on  a  different 
combination  of  months.  The  E-model  was  trained  on  May-June-July  data;  F-model  was 
trained  on  June-July-August  data;  and  G-model  was  trained  on  July-August-September 
data.  Although  neither  a  D-model  nor  an  H-model  was  developed,  the  D,  E,  and 
F-models  would  be  polled  to  produce  a  forecast  for  June;  the  E.  F,  and  G-models  would 
be  polled  for  July;  and  the  F.  G,  and  H-models  would  be  polled  for  August  in  an 
operational  environment. 

The  numbers  241  or  32  In  a  model's  name  Indicate  the  number  of  Input  variables  that 
were  used  In  the  training  mode.  The  32  variables  were  those  that  GOPAD  selected  out 
of  all  the  241  variables  that  were  available  as  candidate  predictor  variables-a  form  of 
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dimensionality  reduction.  Thus,  it  was  possible  to  measure  how  much  information  was 
contained  in  the  variables  that  GOPAD1  (i.e.,  the  BASIC  language  version)  had  selected. 


The  skill  scores  for  these  models  were  calculated  using  the  Lotus™  spreadsheets 
described  earlier  so  they  could  be  compared  to  the  GOPAD-ATLANTA-RER,  the  WSFO, 
and  LFM/MOS  forecast  models. 


Table  12  summarizes  the  three-month  combined  skill  scores.  This  table  shows  that  the 
GOPAD1  selection  process  appears  to  lose  some  information  as  a  result  of 
dimensionality  reduction. 


TABLE  12 

THREE-MONTH  SKILL  SCORES 
USING  BP  - ATLANTA-241  PARAMETER  SETTINGS 


Name 

E-Mod.- . 

£:Msdeis 

G-Models 

Summer 

BP-241 

25.0% 

27.0% 

22.0% 

not  computed 

BP-32 

17.5% 

19.7% 

15.6% 

not  computed 

GOPAD-ATL 

22.2% 

10.0 

18.8% 

26.3* 

LFM/MOS 

n/a 

n/a 

n/a 

16.1% 

WSFO 

n/a 

n/a 

n/a 

21.7% 

BP-241** 

2.1% 

5.9% 

0.79% 

not  computed 

*11113  score  represents  the  result  of  polling  the  E,  F  and  G-models,  and  applying  the  Reliability  Index  Rule 
(RIRm)  as  modified.  In  Phase  I,  the  skill  score  for  the  GO  PAD -ATLANTA-RI R  model  used  22%  as  the 
climatology  for  all  five  months.  In  Phase  II,  the  actual  climatologies  for  each  month  were  computed  and 
used  In  the  computation  of  skill  scores.  Consequently,  there  Is  a  slight  difference  between  the  skill  scores 
for  GO  PAD  reported  In  the  Phase  I  and  Phase  II  final  reports. 


**These  models  used  the  same  OZ  sounding  that  the  LFM/MOS  model  used  to  make  the  12-24Z  forecast; 
whereas  the  other  BP  and  GOPAD  models  used  the  12Z  sounding  to  make  the  12-24Z  forecast 


Table  13  shows  the  skill  scores  for  each  model  for  each  month. 


TABLE  13 

MONTHLY  SKILL  SCORES 

USING  BP-ATLANTA-241  PARAMETER  SETTINGS 

■  Name 

June 

July 

August 

Summer 

BP-E241 

49.3% 

6.9% 

-0.8% 

25.0% 

BP-F241 

46.5% 

12.1% 

7.6% 

27.0% 

BP-G24 

38.1% 

0.5% 

10.8% 

22.0% 

BP-E32 

41.0% 

10.7% 

-23.2% 

17.5% 

BP-F32 

35.4% 

11.1% 

-4.7% 

19.7% 

BP-G32 

24.6% 

7.1% 

4.4% 

15.6% 

GOPAD-RIR 

43.2% 

22.4% 

-0.6% 

26.3% 

LFM/MOS 

26.0% 

12.0% 

3.0% 

16.1% 

WSFO 

35.6% 

11.3% 

9.6% 

21.7% 

Table  14  shows  the  weekly  skill  scores  using  BP-Atlanta-241  parameter  settings, 
rain  column  shows  the  number  of  times  it  rained  during  each  week. 

The 

TABLE  14 

WEEKLY  SKILL  SCORES 

USING  BP-ATLANTA-241  PARAMETER  SETTINGS 

Summer-87  WSFO 

LFM/MOS  GOPAD-E 

GOPAD-F 

GOPAD-G  RIR 

Rain 

June  7 

■19.0 

4.3 

7.6 

-21.7 

15.5 

5.4 

1 

14 

2.7 

14.3 

40.8 

25.2 

35.4 

27.3 

2 

21 

61.3 

39.2 

64.5 

48.4 

40.8 

5.6 

6 

28 

34.1 

18.3 

20.6 

13.7 

47.9 

29.3 

3 

July  5 

24.7 

24.2 

49.0 

1.1 

36.4 

54.9 

3 

12 

•11.3 

2.1 

16.7 

0.7 

25.3 

34.0 

1 

19 

18.5 

1.8 

44.5 

10.3 

39.2 

44.5 

1 

26 

28.5 

17.0 

-24.5 

6.8 

-22.4 

-22.4 

2 

August  2 

6.8 

5.4 

-19.9 

-37.7 

-0.7 

-7.4 

2 

9 

13.9 

11.6 

-1.7 

13.3 

-1.7 

-8.3 

4 

16 

•36.5 

-24.0 

83.6 

-50.6 

15.1 

54.3 

0 

23 

17.0 

52.4 

64.4 

83.6 

67.1 

75.4 

0 

30 

1.9 

-41.7 

-204.0 

-127.3 

-220.4 

-47.0 

0 

Averages 

10.97 

9.61 

10.9 

-2.6 

6.0 

23.5 

J-J-A 

21.7 

16.1 

22.2 

10.0 

18.8 

26.3 

Table  15  shows  weekly  skill  scores  using  BP-Atlanta-241  Parameter  Settings.  The  rain 
column  shows  the  number  of  times  It  rained  during  each  week. 


TABLE  15 

WEEKLY  SKUL  SCORES 

USING  BP-ATLANTA-241  PARAMETER  SETTINGS 

Summer  '  7 

E241 

F241 

G241 

E32 

F32 

G32 

Rain 

June  7 

22.7 

21.2 

10.8 

-5.8 

3.1 

3.2 

1 

14 

31.7 

31.2 

27.5 

38.6 

27.1 

18.6 

2 

21 

65.7 

64.4 

55.1 

62.8 

53.5 

36.7 

6 

28 

46.0 

39.7 

41.3 

33.5 

28.9 

17.0 

3 

July  5 

26.8 

30.1 

-4.7 

24.1 

22.3 

20.3 

O 

u 

12 

-16.9 

11.5 

24.8 

-4.1 

4.3 

5.2 

1 

19 

44.3 

28.9 

34.0 

51.1 

42.4 

35.7 

1 

26 

-11.3 

-9.1 

-19.2 

-3.4 

-3.0 

-13.2 

2 

August  2 

-12.6 

-6.4 

-5.1 

-7.5 

-2.6 

-6.1 

2 

9 

16.2 

22.2 

1.0 

17.6 

24.6 

19.4 

4 

16 

22.0 

-11.1 

53.4 

-50.6 

-64.1 

-28.1 

0 

23 

65.8 

52.6 

72.3 

-46.5 

-14.1 

23.7 

0 

30 

-1.4 

-20.8 

39.7  - 

184.0 

-56.0 

17.7 

0 

Averages 

23.0 

19.6 

25.4 

-5.7 

5.1 

11.5 

J-J-A 

25.0 

27.0 

22.0 

17.5 

19.7 

15.6 

M-J-J-A-S 

16.4 

20.9 

22.4 

11.0 

16.0 

15.8 

3.6.5  Identification  of  Important  Predictor  Variables 

3.6.5. 1  Objective 

The  objectives  of  the  research  are  (1)  to  create  a  list  of  parameters  sorted  according  to 
each  parameter's  relative  weighting  along  the  interconnection  paths  to  the  output  node; 
and  (2)  to  Identify  the  32  GOPAD  selected  parameters  In  this  sorted  list  to  determine  If 
weights  along  the  Interconnections  could  be  used  to  find  Important  variables. 

3.6.5.2  Research  Results 

'Hijo  rsSvSrch  v/ss  conducted  during  April,  1989.  An  investi^sticn.  v/ss  conducted  to 
determine  whether  the  BP  learning  algorithm  could  be  used  to  identify  the  predictor 
variables  that  had  the  greatest  influence  on  the  forecast.  This  effort  was  thought  to  be 
useful  because  the  most  Important  ANS  variables  could  be  compared  to  those  selected 
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by  the  statistical  analysis  process  In  GOPAD1.  Since  statistical  analysis  is  the  most 
time-consuming  GOPAD  process,  It  might  be  possible  for  the  BP  algorithm  to  perform 
dimensionality  reduction  because  It  appeared  to  be  much  faster  than  GOPAD  1. 

The  SP-F241  model  was  used  for  this  research.  The  weights  between  each  Input  node 
and  the  output  node  for  all  interconnection  paths  were  summed.  These  weights  were 
then  sorted.  The  32  candidate  predictor  variables  that  GOPAD  selected  were  identified 
In  this  sorted  list  to  determine  if  they  were  bunched  near  the  top,  middle,  or  bottom  of 
the  list,  or  whether  they  were  scattered  evenly  throughout.  Appendix  A  shows  the 
sorted  list  with  all  241  variables,  and  the  32  variables  that  GOPAD  selected  as 
containing  the  most  statistical  Information.  This  Appendix  shows  that  the  GOPAD- 
selected  variables  are  scattered  fairly  evenly  throughout  this  list.  Table  16  is  an  extract 
from  Appendix  A  showing  just  the  32  variables  that  GOPAD  selected.  Based  on  these 
results,  simply  summing  the  weights  along  the  interconnections  does  not  appear  to 
provide  any  Insight  into  which  variables  are  most  Important. 
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TABLE  16 

SUMMARY  OF  WEIGHTS  ALONG  THE  INTERCONNECTIONS 
FROM  EACH  INPUT  NODE  TO  THE  OUTPUT  NODE 
FOR  THE  GOPAD  SELECTED  VARIABLES 


7 

2.541121 

CA600U 

22 

2.350847 

AA900D 

25 

2.340340 

WA400U 

33 

2.296635 

AA800U 

42 

2.250115 

AA900V 

50 

2.207791 

CA800V 

65 

2.160017 

AA300D 

69 

2.137422 

AA600U 

77 

2.088976 

AA600T 

90 

2.061990 

WA700T 

94 

2.058352 

AA800V 

98 

2.039243 

DAYCNT 

103 

2.031388 

CA900V 

104 

2.030351 

AA100U 

107 

2.024281 

GA400U 

122 

1.982341 

AA700D 

127 

1.974752 

CA800U 

129 

1.961482 

WA800V 

139 

1.938703 

AA700U 

145 

1.930760 

A400U 

155 

1.887024 

AA100T 

158 

1.875487 

CA200T 

165 

1.853136 

A300T 

186 

1.769200 

WA700V 

188 

1.760480 

WA900V 

190 

1.748589 

CA700V 

191 

1.745677 

CA300U 

204 

1.692686 

WA300T 

211 

1.658782 

CA600D 

221 

1.596003 

CA600T 

232 

1.440968 

WA900U 

233 

1.435088 

AA700V 

4.0  GOAL  ORIENTED  PATTERN  DETECTION 


A  1 
*•  * 

Previously.  CCI  had  investigated  expert  systems  for  creating  tactical  weather  forecast 
models.  During  Phase  I  and  II,  standard  and  non-standard  neural  net  approaches  were 
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Investigated.  Through  this  effort,  it  was  found  that  a  proprietary  algorithm,  called  Goal 
Oriented  Pattern  Detection  (GOPAD),  developed  by  Dr.  Kenneth  Young  at  the 
Institute  of  Atmospheric  Physics,  University  of  Arizona,  held  the  greatest  potential  to 
achieve  Army  objectives  for  tactical  weather  forecasting.  During  CCI's  research,  GOPAD 
was  used  to  produce  two  experimental  models-GOPAD-Atlanta-RIR  and  GOPAD  - 
SHOOTOUT-89.  The  GOPAD-Atlanta-RIR  was  a  Probability  of  Precipitation  (PoP)  model 
developed  for  Hartsfield  International  Airport  In  Atlanta,  Georgia.  The  GOPAD- 
SHOOTOUT-89  model  was  a  severe  and  significant  weather  forecast  model  that  was 
independently  tested  by  NOAA  during  a  real-time  severe  and  significant  weather 
forecasting  exercise  called  SHOOTOUT-89  that  took  place  In  the  Boulder  and  Denver, 
Colorado,  area.  The  forecasting  skill  of  these  GOPAD  models  appears  to  be  very 
promising,  In  spite  of  the  limited  number  of  sources  of  candidate  predictor  variables  for 
the  former  model,  and  the  limited  amount  of  Input  data  that  was  available  for  the  latter 
model. 

4.2  GOPAD  Description 

GOPAD  is  technically  described  as  a  statistical,  optimizing,  machine-learning,  analogue, 
forecast  model  creation,  software  tool.  GOPAD  extends  the  multi-discriminant  analysis 
(MDA)  methods  developed  by  Miller  (1962),  and  the  analogue  forecasting  method  of 
Krulzinga  and  Murphy  (1983).  GOPAD  uses  a  k-nearest  neighbor  search.  GOPAD 
produces  data  that  can  be  used  to  construct  a  mesoscale,  real-time  weather  forecasting 
software  program  that  executes  In  seconds. 

In  general,  the  GOPAD  model  development  tool  (1)  creates  optimal  machine-derived 
Indices  from  highly  correlated  variables;  (2)  identifies  the  optimal  statistical 
relationships  between  all  candidate  predictor  variables  and  Indices  using  a  very  large, 
historical  weather  data  set  (i.e.,  100  to  30,000  variables/indices  per  data  point  and  100 
to  40,000  data  points);  and  (3)  optimizes  the  n-space  scaling  and  neighborhood  size  to 
provide  Information  that  can  be  used  to  prepare  a  non-linear  forecast  model.  GOPAD 
models  are  able  to  reveal  the  underlying  physical  relationships  upon  which  the  model  Is 
based  In  the  form  of  exemplars-thus  enabling  a  meteorologist  to  examine  a  model's 
forecast. 

Appendix  B  is  a  paper  that  was  presented  at  the  1989  EOSAEL/TWI  Conference  that 
describes  how  GOPAD  operates,  the  types  of  tactical  mesoscale  forecast  models  that 
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could  be  developed,  a  method  for  defining  forecast  models,  and  how  the  GOPAD 
development  process  Is  used  to  create  new  forecast  models. 

4.3  Conclusions 

The  full  potential  of  the  GOPAD  development  system  to  create  mesoscale  models  that 
can  provide  accurate  forecasting  Information  to  the  Staff  Weather  Officer  on  the 
battlefield  has  not  yet  been  determined.  To  date,  GOPAD  has  been  used  to  produce 
forecast  models  with  very  few  sources  of  candidate  predictor  variables  (Le.,  rawlnsonde 
and  mesonet)  and.  In  Lie  SHOOTOUT-89  exercise,  with  a  severely  limited  and 
inaccurate  historical  data  base.  The  true  potential  of  GOPAD  cannot  be  fully 
documented  until  (a)  all  available  sources  of  predictor  variables  are  used,  (b)  an 
adequate  historical  training  data  base  Is  provided,  and  (3)  accurate  records  are 
available  for  the  event  to  be  predicted.  These  requirements  also  point  out  the  Achilles 
heal  for  any  neural  net  approach.  However,  when  compared  to  the  knowledge 
engineering  problem  for  an  expert  systems  approach  to  tactical  weather  forecasting,  a 
machine  learning  approach  still  appears  to  offer  the  greatest  long  term  cost-benefit  to 
the  Army. 

5.0.  MESOSCALE  WEATHER  FORECASTING  EXERCISE-SHOOTOUT-89 

5.1  Background 

At  the  conclusion  of  Phase  I  research,  GOPAD  1  (Le.,  the  BASIC  language  version) 
appeared  to  be  a  promising  approach  for  automating  the  construction  of  mesoscale 
weather  forecasting  models  for  a  tactical  environment.  However,  GOPAD  1  had  several 
problems  that  appeared  to  limit  Its  usefulness  for  more  complex  problems  that  would 
require  many  sources  of  predictor  variables  and  larger  data  bases.  Consequently, 
research  with  a  more  traditional  neural  net  (i.e.,  the  back-propagation  learning 
algorithm)  was  undertaken  for  comparison  purposes. 

Between  Phase  I  and  Phase  n,  the  GOPAD  1  program  was  upgraded  to  a  VAX  FORTRAN 
version  called  GOPAD2.  This  new  version  appeared  to  overcome  many  of  the  problems 
perceived  In  GOPAD  1.  Since  GOPAD2  appeared  to  have  much  more  potential  than 
GOPAD  1,  a  decision  was  made  to  use  GOPAD2  to  construct  a  model  that  would 
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participate  in  the  Forecast  System  Laboratory/NOAA-sponsored  exercise  called 
SHOOTOUT-89.  The  SHOOTOUT-89  model  is  referred  to  as  GOPAD-RT89. 

5.2  SHOOTOUT-89  Evaluations 

5.2.1  NOAA/FSL  Comparative  Evaluation 

A  massively  co-authored  paper  is  being  prepared  by  NOAA/FSL  that  evaluates  all  six 
models  that  participated  in  the  SHOOTOUT-89  exercise  and  will  be  submitted  for 
publication  in  the  Bulletin  of  American  Meteorological  Society  (BAMS)  in  1990. 

5.2.2  Performance  Evaluation 

Appendix  C  is  a  paper  that  quantitatively  compares  the  performance  of  all  models  that 
participated  in  SHOOTOUT-89  in  more  detail. 

5.3  Conclusions 

There  are  many  different  ways  to  quantitatively  and  qualitatively  compare  the 
performance  of  the  system  that  participated  in  the  SHOOTOUT-89  exercise.  We  choose 
to  summarize  and  caveat  the  quantitative  performance  in.  Table  17  and  Table  18. 

First,  Table  17  shows  the  (Brier)  skill  scores  for  -  *vere  weather  based  upon  the 
forecasts  in  which  there  was  no  statistically  significant  difference  between  reporting  of 
severe  weather  between  1985/87  and  1989  (i.e.,  Regions  n,  m,  and  IV).  This  table 
eliminates  the  effect  from  seriously  under-reporting  significant  weather  in  a)l  regions, 
and  from  seriously  under-reporting  severe  weather  in  region  I.  GOPAD  and  ALPS  are 
most  helped  by  this  caveat. 
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TABLE  17 

SKILL  SCORES  FOR  SEVERE  WEATHER 
IN  REGIONS  D.  m,  AND  IV 


GOPAD  1 

+  2.1% 

GOPAD2 

+  1.9% 

ALPS 

-0.7% 

Willard 

-11.9% 

KASSPr 

-17.0% 

CONVEX 

-24.0% 

OCI 

-32.3% 

Second,  Table  18  shows  the  skill  scores  only  for  region  IV  which  was  the  only  region 
that  had  an  adequate  number  of  events  upon  which  to  train  a  GOPAD  model.  This 
tables  mitigates  the  fact  that  there  was  not  a  large  historical  data  base  available  to 
develop  a  GOPAD  modeL  GOPAD  Is  helped  most  by  this  caveat. 


TABLE  18 

SKILL  SCORES  FOR  SEVERE  WEATHER 
IN  REGION  IV 


GOPAD  1 

+13.8% 

GOPAD2 

+13.2% 

ALPS 

+  1.8% 

KASSPr 

-  0.2% 

OCI 

-3.1% 

Willard 

-12.6% 

CONVEX 

-40.9% 

6.0  CLOUD  IMAGE  REPRESENTATION,  RECOGNITION,  AND  b  >7DERSTANDING 
SOFTWARE 

6.1  Background 

Cloud  Image  Representation,  Recognition,  and  Understanding  Software  (CIRRU8-I} 

autonomously  tracks  Individual  homogeneous  temperature  regions  in  GOES  Infrared 
(IR)  Imagery,  and  Intelligently  derives  cloud  displacement  vectors  from  the  leading  edges 
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of  tracked  cloud-objects.  The  CIRRUS-I  vector  file  could  be  used  as  Input  to  a  vortlcity 
modeling  program  to  produce  synoptic  wind  or  stream  flow  fields  over  North  America. 
The  synoptic  wind  patterns  would  then  be  used  as  an  important  source  of  candidate 
predictor  variables  for  a  neural  net  or  GOPAD. 

6.2  CIRRUS-I  Data  Flow 


Appendix  D  is  a  paper  that  was  presented  at  the  1989  EOSAEL/TWI  Conference.  This 
paper  describes  how  the  CIRRUS  algorithm  operates. 

6.3  CIRRUS-I  Performance 

Appendix  E  is  a  series  of  photographs  that  show  examples  of  how  CIRRUS-I  performs. 

6.4  Conclusions 

CIRRUS-I  was  developed  to  provide  yet  another  source  of  candidate  predictor  variables 
that  would  be  processed  by  GOPAD  to  produce  a  more  accurate  forecast  model.  The 
ultimate  test  of  the  usefulnes?  of  CIRRUS-I  for  mesoscale  forecasting  Is  the  quality  of 
systematic  Information  contained  In  Its  candidate  predictor  variables  that  can 
contribute  to  the  forecast  relative  to  the  Information  available  from  other  sources. 
Therefore,  the  best  way  to  determine  the  value  of  the  CIRRUS-I  and  the  vortlcity 
modeling  program  Is  to  produce  a  multi-year  historical  synoptic  wind  flow  data  base, 
and  measure  the  statistical  contribution  of  CIRRUS-I  as  a  source  of  predictor  variables 
relative  to  the  candidate  predictor  variables  from  other  sources  (e.g.,  rawlnsonde, 
mesonet,  NGM,  barotropic,  etc.). 


CIRRUS-I  lays  the  foundation  for  automating  many  tedious  visual  analysis  tasks  that 
are  performed  by  a  human,  where  shape,  relative  proximity  of  individual  objects  to  each 
other,  a  priori  knowledge,  and  lnferenclng  are  important  for  object  recognition  and 
Image  Interpretation.  In  the  weather  domain,  for  example,  CIRRUS-I  could  be  extended 
to  recognize  features  In  GOES  satellite  Imagery  such  as  fronts,  troughs,  ridges,  and 
areas  of  high  and  low  pressure,  and  to  speculate  about  the  future  state  and  location  of 
these  features. 
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For  other  applications,  CIRRUS-I  demonstrates  the  usefulness  of  a  very  Innovative,  non¬ 
digital,  image  representation  scheme  that  makes  it  possible  to  develop  software 
programs  that  can  use  the  visual  cues  that  a  human  uses  to  recognize  objects.  This 
approach  to  Image  processing  mitigates  many  classical  image  understanding  problems. 

7.0  PHASE  H  RECOMMENDATIONS 

Artificial  Intelligence  techniques  investigated  during  Phase  I  and  II  can  be  used  to  provide  a  new 
level  of  automated  computer  support  to  the  Staff  Weather  Officer.  In  order  to  properly 
evaluate  the  capability  of  these  technologies  to  support  the  SWO,  a  human  forecaster 
who  Is  using  GOPAD  models  and/or  CIRRUS  should  be  directly  compared  to  a  second 
forecaster  who  Is  using  the  next  best  technology  that  might  be  available  in  a  tactical 
environment.  If  it  can  be  shown  that  the  performance  of  the  former  forecaster  1s 
statistically  higher  than  the  latter,  then  the  value  of  a  system  should  be  apparent. 
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BRIER- BASED  SKILL  SCORES  FOR  BP-ATLANTA-241 
USING  OPTIMAL  PARAMETERS 


Optimal  Variable 


Parameters  Changed 

Architectural  Options 

2 

#  Slabs 

30 

#  Hidden  Nodes 

10 

#  Passes 

1 

#  Iterations 

0.01 

Learning  Rate 

0.9 

Momentum 

Table 

Sigmoid  Source 

0.1 

Random  Seed 

241 

#  Input  parameters 

no 

Input  slab  connected  to  output 

Overall  Performance: 

June-July-Aug 

Skill  =  26.1% 

May- June- July-Aug-Sept  Skill  =  21.4% 

Forecast 

Actual 

Day# 

Output 

Rain 

1 

0.201184 

1 

2 

0.499511 

0 

3 

0.305489 

0 

4 

0.345105 

1 

5 

0.062103 

0 

6 

0.209793 

0 

7 

0.242564 

0 

8 

0.231266 

0 

9 

0.080213 

0 

10 

0.203865 

0 

11 

0.528777 

1 

12 

0.494628 

0 

13 

0.460046 

0 

14 

0.736211 

1 

15 

0.650689 

0 

16 

0.540196 

0 

17 

0 .534855 

0 

18 

0.677047 

0 

19 

0.745583 

1 

20 

0.316369 

0 

21 

0.548915 

1 

22 

0.109134 

0 

23 

0.067608 

0 

Squared-Errors 


ANS 

Climatology 

Daily 

0.638 

0.522 

-22.2% 

0.250 

0.077 

-224.2% 

0.093 

0.077 

-21.3% 

0.429 

0.522 

17.9% 

0.004 

0.077 

95.0% 

0.044 

0.077 

42.8% 

0.059 

0.077 

23.5% 

0.053 

0.077 

30.5% 

0.006 

0.077 

91.6% 

0.042 

0.077 

46.0% 

0.222 

0.522 

57.5% 

0.245 

0.077 

-217.9% 

0.212 

0.077 

-175.0% 

0.070 

0.522 

86.7% 

0.423 

0.077 

-450.2% 

0.292 

0.077 

-279.2% 

0.286 

0.077 

-271.8% 

0.458 

0.077 

-495.7% 

0.065 

0.522 

87.6% 

0.100 

0.077 

-30.1% 

0.203 

0.522 

61.0% 

0.012 

0.077 

84.5% 

0.005 

0.077 

94.1% 

;ill  Scores 

Weekly  Monthly  Date 

May  1 


-6.1% 

5.3% 

30.0% 

35.2% 

56.3% 

31.8% 

14.8% 

40.6% 

14.7% 

-5.3% 

-22.4% 

-101.8% 

-26.4% 

-18.5% 

-27.9% 

0.9% 

21.0% 


A-l 


24 

0.458348 

0  0 

.210 

0.077 

-173.0% 

26.3% 

25 

0.054701 

0  0 

.003 

0.077 

96.1% 

58.2% 

26 

0.038394 

0  0 

.001 

0.077 

98.1% 

45.7% 

27 

0.025107 

0  0 

.001 

0.077 

99.2% 

55.8% 

28 

0.050988 

0  0 

.003 

0.077 

96.6% 

56.5% 

29 

0.103930 

0  0 

.011 

0.077 

86.0% 

56.7% 

30 

0.043365 

0  0 

.002 

0.077 

97.6% 

57.2% 

31 

0.077169 

0  0 

.006 

0.077 

92.3% 

95.1% 

12.1% 

32 

0.125600 

0  0 

.016 

0.044 

63.9% 

92.3% 

16.5% 

33 

0.025589 

0  0 

.001 

0.044 

98.5% 

91.9% 

21.3% 

34 

0.175114 

1  0 

.680 

0.626 

-8.7% 

29.7% 

18.3% 

35 

0.167490 

0  0 

.028 

0.044 

35.7% 

24.7% 

18.5% 

36 

0.035914 

0  0 

.001 

0.044 

97.0% 

23.1% 

18.0% 

37 

0.012747 

0  0 

.000 

0.044 

99.6% 

20.5% 

18.3% 

38 

0.031232 

0  0 

001 

0.044 

97.8% 

18.1% 

19.0% 

39 

0.020138 

0  0 

000 

0.044 

99.1% 

19.8% 

19.6% 

40 

0.075721 

0  0. 

006 

0.044 

86.9% 

19.2% 

19.0% 

41 

0.028571 

0  0. 

001 

0.044 

98.1% 

87.7% 

19.3% 

42 

0.227640 

1  0. 

597 

0.626 

4.7% 

31.7% 

12.9% 

43 

0.322525 

0  0. 

104 

0.044 

-138.4% 

20.2% 

15.4% 

44 

0.587706 

1  0. 

170 

0.626 

72.8% 

40.2% 

25.4% 

45 

0.340923 

0  0. 

116 

0.044 

-166.3% 

32.4% 

16.6% 

46 

0.348644 

1  0. 

424 

0.626 

32.2% 

30.9% 

25.5% 

47 

0.605566 

1  0. 

156 

0.626 

75.1% 

40.5% 

35.1% 

48 

0.617858 

1  0. 

146 

0.626 

76.7% 

46.8% 

43.1% 

49 

0.372276 

0  0. 

139 

0.044 

-217.6% 

52.4% 

48.0% 

50 

0.590070 

1  0. 

168 

0.626 

73.1% 

59.0% 

47.2% 

51 

0.606731 

1  0. 

155 

0.626 

75.3% 

59.5% 

50.7% 

52 

0.496337 

1  0. 

254 

0.626 

59.5% 

62.1% 

50.7% 

53 

0.500000 

250 

0.044 

-472.9% 

60.6% 

47.0% 

54 

0.321672 

1  0. 

460 

0.626 

26.5% 

51.2% 

44.8% 

55 

0.205455 

0  0. 

042 

0.044 

3.3% 

44.3% 

46.8% 

56 

0.564338 

1  0. 

190 

0.626 

69.7% 

52.8% 

48.1% 

57 

0.473900 

1  0. 

277 

0.626 

55.8% 

49.4% 

48.2% 

58 

0.084266 

0  0. 

007 

0.044 

83.7% 

43.8% 

48.0% 

59 

0.013363 

0  0. 

000 

0.044 

99.6% 

40.2% 

47.8% 

60 

0.019568 

000 

0.044 

99.1% 

52.4% 

47.7% 

61 

0.401452 

161 

0.044 

-269.3% 

53.9% 

45.6% 

62 

0.516107 

1  0. 

234 

0.563 

58.4% 

56.3% 

46.4% 

63 

0.538740 

290 

0.062 

-366.2% 

32.0% 

43.1% 

64 

0.598312 

1  0. 

161 

0.563 

71.4% 

37.3% 

44.7% 

65 

0.431104 

1  0. 

324 

0.563 

42.5% 

37.8% 

44.4% 

66 

0.385837 

0  0. 

149 

0.062 

-139.1% 

30.6% 

46.7% 

67 

0.324235 

1  0. 

457 

0.563 

18.9% 

26.6% 

45.1% 

68 

0.518790 

O’;";.'-  i 

269 

0.062 

-332.4% 

22.8% 

42.6% 

69 

0.246714 

061 

0.062 

2.2% 

11.8% 

42.0% 

70 

0.124851 

0  0. 

016 

0.062 

75.0% 

25.9% 

42.0% 

71 

n  .nqqqqfi 

0  0. 

pi  n 

n  ns? 

83 .  qa 

i  n  sa 

ao  na- 

72 

0.082845 

007 

0.062 

89.0% 

-3.3% 

42.1% 

73 

0.185061 

034 

0.062 

45.0% 

8.9% 

41.9% 

74 

0.134094 

018 

0.062 

71.1% 

4.8% 

44.5% 

75 

0.200400 

1  0. 

639 

0.563 

-13.5% 

16.2% 

42.1% 

76 

0.049912 

002 

0.062 

96.0% 

22.4% 

40.4% 

77 

0.010091 

000 

0.062 

99.8% 

24.1% 

41.7% 

June  1 


June  30 
July  1 


78 

0.065186 

0 

0.004 

0.062 

93.2% 

24.7% 

42.8% 

79 

0.050237 

0 

0.003 

0.062 

95.9% 

25.2% 

40.7% 

80 

0.066873 

0 

0.004 

0.062 

92.8% 

28.4% 

38.2% 

81 

0.034068 

0 

0.001 

0.062 

98.1% 

30.1% 

40.2% 

82 

0.082919 

0 

0.007 

0.062 

89.0% 

95.0% 

37.7% 

83 

0.056340 

0 

0.003 

0.062 

94.9% 

94.8% 

34.7% 

84 

0.093677 

0 

0.009 

0.062 

85.9% 

92.8% 

32.7% 

85 

0.074432 

0 

0.006 

0.062 

91.1% 

92.5% 

36.9% 

86 

0.070176 

1 

0.865 

0.563 

-53.5% 

4.5% 

29.5% 

87 

0.166810 

1 

0.694 

0.563 

-23.2% 

-10.2% 

25.2% 

88 

0.073164 

0 

0.005 

0.062 

91.4% 

-10.5% 

21.2% 

89 

0.240061 

0 

0.058 

0.062 

7.4% 

-14.0% 

17.1% 

90 

0.095517 

0 

0.009 

0.062 

85.3% 

-14.4% 

17.3% 

91 

0.505859 

0 

0.256 

0.062 

-311.1% 

-31.6% 

12.9% 

92 

0.276299 

1 

0.524 

0.563 

7.0% 

-24.3% 

11.7% 

93 

0.213049 

0 

0.045 

0.052 

13.0% 

-11.5% 

13.8% 

94 

0.209307 

1 

0.625 

'"0.595 

-5.0% 

-4.3% 

7.8% 

95 

0.344002 

1 

0.430 

0.595 

27.7% 

2.3% 

13.1% 

96 

0.338295 

1 

0.438 

0.595 

26.5% 

7.9% 

9.3% 

97 

0.403096 

0 

0.162 

0.052 

-211.5% 

1.4% 

4.4% 

98 

0.419172 

1 

0.337 

0.595 

43.3% 

16.0% 

9.2% 

99 

0.171615 

0 

0.029 

0.052 

43.5% 

18.5% 

8.6% 

100 

0.208662 

1 

0.626 

0.595 

-5.2% 

14.0% 

10.6% 

101 

0.275129 

0 

0.076 

0.052 

-45.1% 

17.3% 

10.2% 

102 

0.081739 

0 

0.007 

0.052 

87.2% 

16.0% 

10.2% 

103 

0.145600 

0 

0.021 

0.052 

59.4% 

13.3% 

9.9% 

104 

0.172868 

0 

0.030 

0.052 

42.7% 

22.4% 

9.4% 

105 

0.103839 

0 

0.011 

0.052 

79.3% 

11.9% 

9.7% 

106 

0.062674 

0 

0.004 

0.052 

92.5% 

14.7% 

9.7% 

107 

0.326808 

0 

0.107 

0.052 

-104.7% 

30.2% 

10.9% 

108 

0.474874 

0 

0.226 

0.052 

-332.3% 

-10.8% 

7.1% 

109 

0.242026 

0 

0.059 

0.052 

-12.3% 

-25.1% 

5.9% 

110 

0.299719 

0 

0.090 

0.052 

-72.2% 

-43.9% 

4.4% 

111 

0.122418 

0 

0.015 

0.052 

71.3% 

-39.8% 

4.0% 

112 

0.049819 

0 

0.002 

0.052 

95.2% 

-37.5% 

3.9% 

113 

0.029958 

0 

0.001 

0.052 

98.3% 

-36.7% 

3.7% 

114 

0.023531 

0 

0.001 

0.052 

98.9% 

-7.6% 

3.7% 

115 

0.122103 

0 

0.015 

0.052 

71.4% 

50.1% 

3.3% 

116 

0.141637 

0 

0.020 

0.052 

61.5% 

60.6% 

2.9% 

117 

0.279434 

0 

0.078 

0.052 

-49.7% 

63.9% 

1.6% 

118 

0.108471 

0 

0.012 

0.052 

77.4% 

64.7% 

8.0% 

119 

0.116060 

0 

0.013 

0.052 

74.2% 

61.7% 

12.2% 

120 

0.241310 

0 

0.058 

0.052 

-11.6% 

46.0% 

11.0% 

121 

0.447218 

0 

0.200 

0.052 

-283.4% 

-8.6% 

7.9% 

122 

0.104385 

0 

0.011 

0.052 

79.1% 

-7.5% 

7.7% 

123 

0.548190 

0 

0.301 

0.052 

-476.1% 

-84.3% 

6.6% 

124 

0.397470 

0 

0.158 

0.053 

-196.1% 

-105.5% 

4.1% 

IOC 

n  -7r\AAKA 

V  *  s/  V  4  A  A 

n 

V 

0.093 

0.053 

-73.7% 

-126.8% 

3 . 0% 

126 

0.109419 

0 

6 !  012 

6.053 

77.6% 

-125.7% 

5.3% 

127 

0.106783 

0 

0.011 

0.053 

78.6% 

-112.3% 

2.4% 

128 

0.341143 

1 

0.434 

0.591 

26.6% 

-12.1% 

2.4% 

129 

0.511472 

1 

0.239 

0.591 

59.6% 

13.9% 

14.2% 

130 

0.424650 

0 

0.180 

0.053 

-237.9% 

22.2% 

4.8% 

131 

0.119716 

0 

0.014 

0.053 

73.1% 

32.2% 

5.3% 

July  31 
Aug  1 


Aug  31 
Sept  1 


A- 3 


132 

0.280616 

0 

0.079 

0.053 

133 

0.317637 

0 

0.101 

0.053 

134 

0.717210 

0 

0.514 

0.053 

135 

0.637933 

0 

0.407 

0.053 

136 

0.294620 

0 

0.087 

0.053 

137 

0.077867 

0 

0.006 

0.053 

138 

0.065246 

0 

0.004 

0.053 

139 

0.388849 

0 

0.151 

0.053 

140 

0.347979 

0 

0.121 

0.053 

141 

0.077378 

0 

0.006 

0.053 

142 

0.660391 

1 

0.115 

0.591 

143 

0.147309 

0 

0.022 

0.053 

144 

0.042324 

0 

0.002 

0.053 

145 

0.194062 

0 

0.038 

0.053 

146 

0.068787 

0 

0.005 

0.053 

147 

0.070240 

0 

0.005 

0.053 

148 

0.008985 

0 

0.000 

0.053 

149 

0.009869 

0 

0.000 

0.053 

150 

0.128415 

0 

0.016 

0.053 

151 

0.169818 

0 

0.029 

0.053 

152 

0.283186 

0 

0.080 

0.053 

153 

0.110087 

0 

0.012 

0.053 

-47.6% 

33.1% 

-89.1% 

27.0% 

-864.0% 

-7.7% 

-662.7% 

-68.3% 

-62.7% 

-270.1% 

88.6% 

-223.5% 

92.0% 

-220.8% 

-183.4% 

-240.2% 

-126.9% 

-245.6% 

88.8% 

-109.5% 

80.5% 

46.2% 

59.3% 

53.3% 

96.6% 

53.8% 

29.4% 

50.1% 

91.1% 

66.2% 

90.8% 

78.9% 

99.8% 

79.6% 

99.8% 

81.0% 

69.1% 

82.4% 

46.0% 

75.1% 

-50.3% 

63.8% 

77.3% 

61.8% 

6.5% 

5.7% 

-12.7% 

-26.6% 

-28.6% 

-28.4% 

-28.4% 

-29.9% 

-26.1% 

-24.1% 

-4.7% 

-4.8% 

-4.8% 

-5.8% 

-5.9% 

-5.6% 

-4.9% 

-2.6% 

-2.7% 

-3.1% 

-3.7% 

2.0%  Sept  30 


A- 4 


BRIER-BASED  SKILL  SCORES  FOR  BP-ATLANTA-241 
USING  LEARNING  RATE  =  .005 


Optimal 

Parameters 

Variable 

Changed 

Architectural  Options 

2 

#  Slabs 

30 

#  Hidden  Nodes 

10 

#  Passes 

1 

#  Iterations 

0.01 

.005 

Learning  Rate 

0.9 

Momentum 

Table 

Sigmoid  Source 

0.1 

Random  Seed 

241 

#  Input  parameters 

no 

Input  slab  connected  to  output 

Overall  Performance: 


June-July-Aug  Skill  =  23.9% 

May- June- July-Aug-Sept  Skill  =  19.0% 


Day# 


Squared-Errors 

Forecast  Actual  === 

Output  Rain  ANS  Climatology  Daily 


Skill  Scores 
Weekly  Monthly 


0.522 


-25.8% 


1 

19% 

1 

0.657 

2 

46% 

0 

0.213 

3 

28% 

0 

0.077 

4 

31% 

1 

0.477 

5 

10% 

0 

0.011 

6 

19% 

0 

0.035 

7 

22% 

0 

0.050 

8 

29% 

0 

0.085 

9 

10% 

0 

0.009 

10 

19% 

0 

0.036 

11 

42% 

1 

0.333 

12 

52% 

> 

0.267 

13 

44% 

0 

0.192 

14 

67% 

1 

0.108 

15 

57% 

0 

0.321 

16 

54% 

0 

0.292 

17 

50% 

0 

0.251 

18 

58% 

0 

0.334 

19 

71% 

1 

0.081 

20 

41% 

0 

0.166 

21 

57% 

1 

0.183 

22 

17% 

0 

0.028 

23 

15% 

0 

0.022 

24 

47% 

0 

0.226 

0.077 

-176.5% 

0.077 

0.5% 

0.522 

8.6% 

0.077 

86.1% 

0.077 

54.6% 

0.077 

35.1% 

-6.3% 

0.077 

-10.9% 

3.7% 

0.077 

87.7% 

24.4% 

0.077 

52.7% 

28.4% 

0.522 

36.2% 

43.1% 

0.077 

-246.5% 

17.1% 

0.077 

-149.4% 

1.1% 

0.522 

79.4% 

27.9% 

0.077 

-317.0% 

11.4% 

0.077 

-279.2% 

-8.3% 

0.077 

-225.8% 

-23.3% 

0.077 

-333.4% 

-79.2% 

0.522 

84.4% 

-10.4% 

0.077 

-115.1% 

-8.6% 

0.522 

64.9% 

-13.9% 

0.077 

63.2% 

6.6% 

0.077 

71.2% 

25.5% 

0.077 

-193.1% 

27.3% 

Date 
May  1 
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25 

13% 

0 

0.018 

0.077 

26 

9% 

0 

0.008 

0.077 

27 

7% 

0 

0.005 

0.077 

28 

11% 

0 

0.012 

0.077 

29 

19% 

0 

0.035 

0.077 

30 

6% 

0 

0.004 

0.077 

31 

19% 

0 

0.035 

0.077 

32 

19% 

0 

0.036 

0.044 

33 

7% 

0 

0.005 

0.044 

34 

25% 

1 

0.560 

0.626 

35 

28% 

0 

0.081 

0.044 

36 

8% 

0 

0.006 

0.044 

37 

2% 

0 

0.000 

0.044 

38 

5% 

C 

0.002 

0.044 

39 

4% 

0 

0.002 

0.044 

40 

11% 

0 

0.011 

0.044 

41 

6% 

0 

0.004 

0.044 

42 

39% 

1 

0.373 

0  626 

43 

32% 

0 

0.105 

0.044 

44 

56% 

1 

0.196 

0.626 

45 

46% 

0 

0.211 

0.044 

46 

31% 

1 

0.478 

0.626 

47 

59% 

1 

0.170 

0.626 

48 

58% 

1 

0.175 

0.626 

49 

49% 

0 

0.237 

0.044 

50 

57% 

1 

0.186 

0.626 

51 

63% 

1 

0.138 

0.626 

52 

58% 

1 

0.177 

0.626 

53 

54% 

0 

0.294 

0.044 

54 

31% 

1 

0.471 

0.626 

55 

11% 

0 

0.012 

0.044 

56 

48% 

1 

0.272 

0.626 

57 

46% 

1 

0.293 

0.626 

58 

19% 

0 

0.035 

0.044 

59 

5% 

0 

0.002 

0.044 

60 

4% 

0 

0.002 

0.044 

61 

47% 

0 

0.219 

0.044 

62 

51% 

1 

0.240 

0.563 

63 

57% 

0 

0.321 

0.062 

64 

53% 

1 

0.217 

0.563 

65 

44% 

1 

0.310 

0.563 

66 

39% 

0 

0.150 

0.0  62 

67 

28% 

1 

0.520 

0.563 

68 

50% 

0 

0.247 

0.062 

69 

32% 

0 

0.101 

0.062 

70 

22% 

0 

0.049 

0.062 

71 

20% 

0 

0.040 

0.062 

H  O 
/  ^ 

1  OO. 
-L^T> 

A 

\J 

a  r\  *i  a 

VJ  ♦ 

r\  a  /*a 
U  •  U 

73 

14% 

0 

0.019 

0.062 

74 

12% 

0 

0.013 

0.062 

75 

20% 

1 

0.634 

0.563 

76 

8% 

0 

0.007 

0,062 

77 

2% 

0 

0.000 

0.0C2 

78 

6% 

0 

0.004 

0.C62 
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76.8%  49.3% 

90.0%  33.9% 

93.7%  50.2% 

83.9%  40.8% 

54.1%  39.5% 

95.4%  43.0% 

53.9%  78.2%  9.6%  May  31 

16.5%  73.1%  13.7%  June  1 

88.7%  71.8%  17.7% 

10.5%  32.6%  17.1% 

-85.6%  23.4%  17.0% 

85.6%  23.7%  16.5% 

98.9%  21.3%  16.7% 

94.4%  22.1%  17.1% 

95.8%  26.0%  18.4% 

73.7%  25.2%  17.7% 

91.5%  64.9%  17.8% 

40.5%  55.1%  18.8% 

-139.6%  44.0%  21.8% 

68.6%  52.9%  30.3% 

-382.9%  38.7%  20.7% 

23.6%  32.9%  26.1% 

72.9%  41.7%  35.5% 

72.0%  46.9%  42.4% 

-444.2%  40.3%  43.6% 

70.3%  48.6%  42.9% 

77.9%  50.4%  47.9% 

71.8%  58.9%  48.8% 

-573.5%  57.2%  44.7% 

24.8%  47.8%  42.7% 

71.4%  42.5%  45.3% 

56.5%  51.8%  45.9% 

53.2%  48.5%  46.1% 

19.1%  41.0%  45.5% 

95.2%  32.8%  45.4% 

95.7%  47.0%  45.6% 

-401.0%  43.1%  42.8%  June  30 

57.5%  46.6%  43.7%  July  1 

-415.1%  22.1%  40.4% 

61.5%  24.1%  41.8% 

45.0%  30.5%  41.8% 

-140.6%  23.4%  42.6% 

7.7%  18.4%  41.2% 

-296.1%  17.9%  38.9% 

-62.9%  3.8%  38.0% 

21.1%  17.8%  37.7% 

35.0%  1.4%  37.4% 

'in  AO  «  /S  n  M  a  ^  n 

/  /  .  3-5  -13  .  ^  /  .  OiS 

70.1%  -5.7%  37.5% 

78.4%  -10.9%  37.5% 

-12.5%  7.1%  35.5% 

88.9%  17.2%  33.6% 

99.5%  22.4%  36.0% 

94.0%  26.3%  37.3% 


79 

8% 

0 

0.006 

0.062 

90.6% 

27.1% 

35.0% 

80 

12% 

0 

0.013 

0.062 

78.6% 

27.7% 

32.3% 

81 

8% 

0 

0.006 

0.062 

90.6% 

28.5% 

35.5% 

82 

14% 

0 

0.018 

0.062 

70.4% 

87.5% 

32.8% 

83 

13% 

0 

0.016 

0.062 

74.4% 

85.4% 

28.9% 

84 

17% 

0 

0.029 

0.062 

53.2% 

78.8% 

24.7% 

85 

16% 

0 

0.025  * 

0.062 

59.5% 

73.9% 

29.4% 

86 

10% 

1 

0.806 

0.563 

-43.0% 

2.5% 

23.1% 

87 

17% 

1 

0.685 

0.563 

-21.5% 

-10.2% 

18.9% 

88 

9% 

0 

0.008 

0.062 

86.8% 

-10.4% 

15.7% 

89 

27% 

0 

0.072 

0.062 

-15.2% 

-14.1% 

11.0% 

90 

10% 

0 

0.010 

0.062 

83.2% 

-13.7% 

11.8% 

91 

40% 

0 

0.159 

0.062 

-155.0% 

-22.7% 

9.2% 

92 

24% 

1 

0.581 

0.563 

-3.1% 

-19.7% 

7.4% 

July  31 

93 

24% 

0 

0.060 

0.052 

-14.3% 

-10.3% 

10.2% 

Aug  1 

94 

27% 

1 

0.535 

0.595 

10.1% 

2.4% 

5.8% 

95 

34% 

1 

0.442 

0.595 

25.7% 

6.7% 

11.6% 

96 

28% 

1 

0.514 

0.595 

13.6% 

0.9% 

7.5% 

97 

37% 

0 

0.140 

0.052 

-168.3% 

3.4% 

2.5% 

98 

39% 

1 

0.369 

0.595 

38.0% 

13.4% 

7.1% 

99 

25% 

0 

0.064 

0.052 

-23.4% 

16.3% 

6.8% 

100 

19% 

1 

0.649 

0.595 

-9.0% 

11.9% 

8.2% 

101 

24% 

0 

0.056 

0.052 

-7.1% 

12.0% 

8.7% 

102 

10% 

0 

0.011 

0.052 

79.2% 

9.6% 

9.2% 

103 

19% 

0 

0.038 

0.052 

27.7% 

8.6% 

9.1% 

104 

20% 

0 

0.039 

0.052 

24.9% 

15.5% 

8.6% 

105 

15% 

0 

0.023 

0.052 

56.4% 

3.2% 

8.4% 

106 

11% 

0 

0.013 

0.052 

75.7% 

8.9% 

8.2% 

107 

24% 

0 

0.057 

0.052 

-10.1% 

35.2% 

10.0% 

108 

36% 

0 

0.128 

0.052 

-146.0% 

15.4% 

7.8% 

109 

24% 

0 

0.059 

0.052 

-13.5% 

2.2% 

6.7% 

110 

27% 

0 

0.072 

0.052 

-37.5% 

-7.2% 

5.4% 

111 

24% 

0 

0.057 

0.052 

-9.8% 

-12.1% 

4.4% 

112 

10% 

0 

0.010 

0.052 

80.0% 

-8.7% 

4.3% 

113 

5% 

0 

0.002 

0.052 

95.5% 

-5.9% 

4.2% 

114 

5% 

0 

0.003 

0.052 

95.0% 

9.1% 

4.3% 

115 

11% 

0 

0.012 

0.052 

77.4% 

41.0% 

4.2% 

116 

14% 

0 

0.021 

0.052 

59.9% 

51.5% 

4.2% 

117 

24% 

0 

0.060 

0.052 

-14.8% 

54.8% 

3.4% 

118 

16% 

0 

0.025 

0.052 

52.7% 

63.7% 

8.7% 

119 

20% 

0 

0.041 

0.052 

21.4% 

55.3% 

12.3% 

120 

24% 

0 

0.059 

0.052 

-12.3% 

39.9% 

11.1% 

121 

41% 

0 

0.166 

0.052 

-219.2% 

-5.0% 

9.0% 

122 

21% 

0 

0.045 

0.052 

14.2% 

-14.0% 

8.1% 

123 

53% 

0 

0.278 

0.052 

-432.0% 

-84.3% 

5.5% 

Aug  31 

124 

46% 

0 

0.216 

0.053 

-304.2% 

-126.2% 

2.8% 

Sept  1 

125 

29% 

0 

0.084 

0.053 

-57.6% 

-141.6% 

2.3% 

126 

16% 

o 

0.025 

0 . 053 

53 . 6% 

-T5C  £3- 

d.  V  V  «  s/ 

1  &9- 

4.  •  V  V 

127 

17% 

6 

6.030 

0.053 

43.6% 

-128.0% 

-1.9% 

128 

34% 

1 

0.434 

0.591 

26.7% 

-22.1% 

0.5% 

129 

45% 

1 

0.301 

0.591 

49.1% 

5.6% 

10.2% 

130 

38% 

0 

0.143 

0.053 

-168.7% 

15.0% 

2.3% 

131 

13% 

0 

0.017 

0.053 

67.5% 

28.7% 

3.8% 

132 

24% 

0 

0.057 

0.053 

-5.9% 

30.6% 

6.4% 
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133 

29% 

0 

134 

68% 

0 

135 

61% 

0 

136 

30% 

0 

137 

11% 

0 

138 

9% 

0 

139 

38% 

0 

140 

40% 

0 

141 

10% 

0 

142 

62% 

1 

143 

22% 

0 

144 

6% 

0 

145 

18% 

0 

146 

11% 

0 

147 

7% 

0 

148 

2% 

0 

149 

2% 

0 

150 

?  7% 

0 

151 

17% 

0 

152 

24% 

0 

153 

20% 

0 

0.082  0.053 
0.464  0.053 
0.372  0.053 
0.092  0.053 
0.012  0.053 
0.008  0.053 
0.143  0.053 
0.161  0.053 
0.011  0.053 
0.142  0.591 
0.047  0.053 
0.004  0.053 
0.033  0.053 
0.012  0.053 
0.006  0.053 
0.000  0.053 
0.001  0.053 
0.028  0.053 
0.030  0.053 
0.060  0.053 
0.039  0.053 


-53.9% 

26.6% 

-768.8% 

-3.3% 

-596.2% 

-57.5% 

-71.8% 

-228.3% 

77.5% 

-193.1% 

84.7% 

-190.6% 

-167.1% 

-213.7% 

-201.7% 

-234.8% 

79.6% 

-113.6% 

76.0% 

37.7% 

12.7% 

42.6% 

93.0% 

43.5% 

37.2% 

40.7% 

77.8% 

55.1% 

89.6% 

72.1% 

99.3% 

73.3% 

98.8% 

72.6% 

47.8% 

77.6% 

43.2% 

70.5% 

-11.6% 

63.6% 

26.3% 

56.2% 

5.5% 

-10.9% 

-23.0% 

-24.8% 

-24.4% 

-24.1% 

-27.2% 

-28.3% 

-26.5% 

-8.0% 

-7.6% 

-7.4% 

-8.3% 

-8.5% 

-8.3% 

-7.6% 

-5.8% 

-5.9% 

-5.5% 

-5.5% 

-1.6%  Sept  30 


I 

I 

I 


B 

I 

I 

I 


■ 

I 

I 

I 

I 
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BRIER-BASED  SKILL  SCORES  FOR  BP -ATLANTA- 241 
USING  NUMBER  OF  PASSES  =  20 


Optimal 

Parameters 

Variable 

Changed 

Architectural  Options 

2 

#  Slabs 

30 

#  Hidden  Nodes 

10 

20 

#  Passes 

1 

#  Iterations 

0.01 

Learning  Rate 

0.9 

Momentum 

Table 

Sigmoid  Source 

0.1 

Random  Seed 

241 

#  Input  parameters 

no 

Input  slab  connected  to  output 

Overall  Performance: 


June-July-Aug  Skill  = 

May-June-July-Aug-Sept  Skill  = 


19.8% 

19.5% 


Forecast 


Squared-Errors 

Actual  ==================  ==== 


Skill  Scores 


Hi 

Output 

Rain 

ANS 

Climatology 

Daily 

Weekly 

1 

0.185356 

1 

0.664 

0.522 

-27.1% 

2 

0.548190 

0 

0.301 

0.077 

-290.5% 

3 

0.299924 

0 

0.090 

0.077 

-16.9% 

4 

0.410637 

1 

0.347 

0.522 

33.5% 

5 

0.020944 

0 

0.000 

0.0*77 

99.4% 

6 

0.125600 

0 

0.016 

0.077 

79.5% 

7 

0.255900 

0 

0.065 

0.077 

14.9% 

-3.8% 

8 

0.091058 

0 

0.008 

0.077 

89.2% 

15.9% 

9 

0.052037 

0 

0.003 

0.077 

96.5% 

46.1% 

10 

0.200088 

0 

0.040 

0.077 

48.0% 

51.2% 

11 

0.637933 

1 

0.131 

0.522 

74.9% 

73.2% 

12 

0.426560 

0 

0.182 

0.077 

-136.5% 

54.7% 

13 

0.343781 

0 

0.118 

0.077 

-53.6% 

44.3% 

14 

0.617397 

1 

0.146 

0.522 

72.0% 

56.0% 

15 

0.561455 

0 

0.315 

0.077 

-309.7% 

34.5% 

16 

0.487551 

0 

0.238 

0.077 

-208.9% 

18.1% 

17 

0.319970 

0 

0.102 

0.077 

-33.0% 

13.7% 

«  A 
10 

0 . 655397 

0 

0.430 

0.077 

-459.2% 

-5t.7% 

19 

0.683208 

1 

0.100 

0.522 

80.8% 

-1.5% 

20 

0.056236 

0 

0.003 

0.077 

95.9% 

6.5% 

21 

0.471466 

1 

0.279 

0.522 

46.5% 

-2.8% 

22 

0.027480 

0 

0.001 

0.077 

99.0% 

19.2% 

23 

0.020274 

0 

0.000 

0.077 

99.5% 

35.8% 

24 

0.497070 

0 

0.247 

0.077 

-221.1% 

25.7% 

Date 
May  1 


25 

0.020824 

0 

0.000 

0.077 

26 

0.005905 

0 

0.000 

0.077 

27 

0.007232 

0 

0.000 

0.077 

28 

0.020884 

0 

0.000 

0.077 

29 

0.036459 

0 

0.001 

0.077 

30 

0.012206 

0 

0.000 

0.077 

31 

0.013299 

0 

0.000 

0.077 

32 

0.050424 

0 

0.003 

0.044 

33 

0.017934 

0 

0.000 

0.044 

34 

0.217836 

1 

0.612 

0.626 

35 

0.058131 

0 

0.003 

0.044 

36 

0.016308 

0 

0.000 

0.044 

37 

0.001824 

0 

0.000 

0.044 

38 

0.012883 

0 

O.OOG 

0.044 

39 

0.009312 

0 

0.000 

0.044 

40 

0.037856 

0 

0.001 

0.044 

41 

0.011008 

0 

0.000 

0.044 

42 

0.180106 

1 

0.672 

0.626 

43 

0.453019 

0 

0.205 

0.044 

44 

0.633183 

1 

0.135 

0.626 

45 

0.228328 

0 

0.052 

0.044 

46 

0.199619 

1 

0.641 

0.626 

47 

0.587470 

1 

0.170 

0.626 

48 

0.736400 

1 

0.069 

0.626 

49 

0.177668 

0 

0.032 

0.044 

50 

0.649801 

1 

0.123 

0.626 

51 

0.594080 

1 

0.165 

0.626 

52 

0.310903 

1 

0.475 

0.626 

53 

0.420361 

0 

G.177 

0.044 

54 

0.212395 

1 

0.620 

0.626 

0.154184 

0 

0.024 

0.044 

56 

0.588416 

1 

0.169 

0.626 

57 

0.280813 

1 

0.517 

0.626 

*8 

0.019700 

0 

0.000 

0.044 

52 

0.003090 

0 

0.000 

0.044 

60 

0.007815 

0 

0.000 

0.044 

61 

0.338076 

0 

0.114 

0.044 

62 

0.488527 

1 

0.262 

0.563 

63 

0.484623 

0 

0.235 

0.062 

64 

0.722527 

1 

0.077 

0.563 

65 

0.437342 

1 

0.317 

0.563 

66 

0.423457 

0 

0.179 

0.062 

67 

0.332201 

1 

0.446 

0.563 

68 

0.555435 

0 

0.309 

0.062 

69 

0.155590 

0 

0.024 

0.062 

70 

0.074298 

0 

0.006 

0.062 

71 

0.045777 

0 

0.002 

0.062 

72 

0.037047 

0 

0.001 

0.062 

73 

0.181118 

0 

0.033 

0.062 

74 

0.095181 

0 

0.009 

0.062 

75 

0.078925 

1 

0.848 

0.563 

76 

0.021125 

0 

0.000 

0.062 

77 

0.005992 

0 

0.000 

0.062 

78 

0.084795 

0 

0.007 

0.062 

99.4%  55.8% 

100.0%  46.0% 

99.9%  46.3% 

99.4%  53.7% 

98.3%  53.6% 

99.8%  53.7% 

99.8%  99.5%  24.2%  May  31 

94.2%  99. 1%  30.8%  June  1 

99.3%  98.9%  36.9% 

2.2%  39.6%  33.4% 

92.3%  37.3%  34.0% 

99.4%  35.2%  33.5% 

100.0%  32.8%  33.3% 

99.6%  30.3%  34.3% 

99.8%  30.6%  34.0% 

96.7%  30.5%  33.5% 

99.7%  98.2%  33.9% 

-7.4%  24.0%  23.5% 

-370.3%  0.9%  22.4% 

78.5%  31.0%  30.5% 

-19.5%  27.5%  25.3% 

-2.4%  16.8%  27.0% 

72.8%  28.8%  35.2% 

88.9%  39.5%  41.5% 

27.7%  50.5%  47.6% 

80.4%  62.0%  48.2% 

73.7%  61.1%  50.0% 

24.1%  55.9%  47.9% 

-304.9%  62.4%  45.1% 

0.9%  48.4%  40.8% 

45.5%  38.7%  43.6% 

72.9%  45.5%  45.3% 

17.4%  33.2%  42.8% 

99.1%  24.7%  42.5% 

100.0%  26.5%  42.3% 

99.9%  35.1%  42.1% 

-161.9%  43.9%  40.5%  June  30 

53.6%  46.6%  41.3%  July  1 

-277.3%  20.9%  38.6% 

86.3%  49.5%  41.2% 

43.8%  46.7%  41.1% 

-188.1%  37.7%  42.2% 

20.8%  32.7%  40.7% 

-395.6%  25.2%  37.7% 

61.1%  18.2%  37.6% 

91.1%  30.0%  37.7% 

96.6%  10.8%  37.8% 

97.8%  -3.2%  37.9% 

47.3%  12.4%  37.7% 

85.4%  12.0%  41.0% 

-50.6%  1.4%  37.6% 

99.3%  4.0%  35.3% 

99.9%  4.5%  36.0% 

88.4%  4.0%  39.0% 


A-10 


79 

0.023757 

0 

0.001 

0.062 

99.1% 

4.1% 

36.9% 

80 

0.015573 

0 

0.000 

0.062 

99.6% 

7.6% 

33.3% 

81 

0.007800 

0 

0.000 

0.062 

99.9% 

8.5% 

33.8% 

82 

0.032191 

0 

0.001 

0.062 

98.3% 

97.8% 

30.3% 

83 

0.018404 

0 

0.000 

0.062 

99.5% 

97.8% 

26.9% 

84 

0.045310 

0 

0.002 

0.062 

96.7% 

97.4% 

27.9% 

85 

0.029367 

0 

0.001 

0.062 

98.6% 

98.8% 

31.0% 

86 

0.031828 

1 

0.937 

0.563 

-66.4% 

-0.6% 

25.0% 

87 

0.118895 

1 

0.776 

0.563 

-37.8% 

-19.5% 

19.5% 

88 

0.037116 

0 

0.001 

0.062 

97.8% 

-19.6% 

14.7% 

89 

0.138694 

0 

0.019 

0.062 

69.1% 

-20.9% 

15.0% 

90 

0.032405 

0 

0.001 

0.062 

98.3% 

-20.9% 

15.3% 

91 

0.581778 

0 

0.338 

0.062 

-443.7% 

-44.3% 

9.4% 

92 

0.280813 

1 

0.517 

0.563 

8.2% 

-33.6% 

8.6% 

93 

0.153549 

0 

0.024 

0.052 

54.8% 

-17.5% 

10.3% 

94 

0.130174 

1 

0.757 

0.595 

-27.1% 

-13.5% 

2.5% 

95 

0.187578 

1 

0.660 

0.595 

-10.9% 

-16.2% 

4.0% 

96 

0.356670 

1 

0.414 

0.595 

30.5% 

-7.3% 

-0.7% 

97 

0.464658 

0 

0.216 

0.052 

-313.9% 

-16.3% 

-7.5% 

98 

0.311740 

1 

0.474 

0  .595 

20.4% 

-0.4% 

-3.3% 

99 

0.061480 

0 

0.004 

0.052 

92.8% 

-0.4% 

-4.7% 

100 

0.134775 

1 

0.749 

0.595 

-25.7% 

-6.2% 

-2.9% 

101 

0.202285 

0 

0.041 

0.052 

21.6% 

-0.7% 

-3.3% 

102 

0,033399 

0 

0.001 

0.052 

97.9% 

4.9% 

-3.4% 

103 

0.075448 

0 

0.006 

0.052 

89.1% 

-2.6% 

-3.6% 

104 

0.083890 

0 

0,007 

0.052 

86.5% 

11.8% 

-3.9% 

105 

0.045310 

0 

0.002 

0.052 

96.1% 

10.9% 

-3.5% 

106 

0.026455 

0 

0.001 

0.052 

98.7% 

11.3% 

-3.6% 

107 

0.226099 

0 

0.051 

0.052 

2.0% 

70.2% 

0.8% 

108 

0.583678 

0 

0.341 

0.052 

-553.1% 

-11.8% 

-4.9% 

109 

0.269517 

0 

0.073 

0.052 

-39.2% 

-31.4% 

-6.3% 

110 

0.308815 

0 

0.095 

0.052 

-82.8% 

-56.0% 

-8.0% 

111 

0.026205 

0 

0.001 

0.052 

98.7% 

-54.2% 

-8.1% 

112 

0.010997 

0 

0.000 

0.052 

99.8% 

-53.7% 

-8.3% 

113 

0.013808 

0 

0.000 

0.052 

99.6% 

-53.6% 

-8.5% 

114 

0.009831 

0 

0.000 

0.052 

99.8% 

-39.6% 

-8.7% 

115 

0.114467 

0 

0.013 

0.052 

74.9% 

50.1% 

-9.1% 

116 

0.165727 

0 

0.027 

0.052 

47.3% 

62.5% 

-9.7% 

117 

0.386763 

0 

0.150 

0.052 

-186.7% 

47.6% 

-12.4% 

118 

0.070367 

0 

0.005 

0.052 

90.5% 

46.5% 

-5.8% 

119 

0.069163 

0 

0.005 

0.052 

90.8% 

45.2% 

-1.1% 

120 

0.143067 

0 

0.020 

0.052 

60.8% 

39.6% 

-1.7% 

121 

0.404036 

0 

0.163 

0.052 

-212.9% 

-5.0% 

-4.8% 

122 

0.092114 

0 

0.008 

0.052 

83.7% 

-3.8% 

-5.2% 

123 

0.493164 

0 

0.243 

0.052 

-366.2% 

-62.9% 

-3.5% 

124 

0.418696 

0 

0.175 

0.0^' 

-228.5% 

-69.4% 

-7.7% 

125 

0.177810 

0 

0.032 

0 . 05  j 

40.7% 

-76.1% 

-7.9% 

1  O  C 

J.  V 

A  A  C  /TOO  C 

V  •  \J  V 

A 

V 

A  A  A  A 

V  «  w  -x 

A  A  C  O 

V  • 

r\  i  no. 

3  X  ♦  O'O 

^  tr  ao 
”  /  O  ♦  ‘I'O 

A  r  o 
~-J  •  >->' 0 

127 

0.056340 

0 

0.003 

0.053 

94.1% 

-70.1% 

-0.6% 

128 

0.282591 

1 

0.515 

0.591 

13.0% 

-7.9% 

-3.8% 

129 

0.518546 

1 

0.232 

0.591 

60.8% 

16.9% 

10.4% 

130 

0.328960 

0 

0.108 

0.053 

-102.8% 

26.2% 

6.7% 

131 

0.112697 

0 

0.013 

0.053 

76.2% 

37.5% 

6.5% 

132 

0.371820 

0 

0.138 

0.053 

-159.1% 

30.1% 

10.2% 

July  31 
Aug  1 


Aug  31 
Sept  1 


A- 11 


133 

0.314050 

0 

0.0 

134 

0.791821 

0 

0.6 

135 

0.688259 

0 

0.4 

136 

0.221350 

0 

0.0 

137 

0.110183 

0 

0.0 

138 

0.035310 

0 

0.0 

139 

0.353762 

0 

0.1 

140 

0.335677 

0 

0.1 

141 

0.039158 

0 

0.0 

3  42 

0.769080 

1 

0.0. 

143 

0.090654 

0 

0.0 

144 

0.040313 

0  0.01 

145 

0.176387 

0  0.0: 

146 

0.032805 

0 

0.01 

147 

0.035577 

0 

o.oi 

148 

0.001396 

0 

0.01 

149 

0.000851 

0 

0.0! 

150 

0.049634 

0 

0.01 

151 

0.184179 

0 

o.o: 

152 

0.210766 

0 

0.0- 

153 

0.027350 

0 

0.01 

Sept  30 


I 


I 

I 

I 

I 


I 

I 


NWSFO  POP  FORECAST  SKILL  SCORE 
0-12  HRS  POP  FORECASTER  FOR  HARTSFIELD  INTERNATIONAL  AIRPORT 


June- July -August  Skill  Score  =  22.2% 


Squared-Errors  Skill  Scores 

Forecast  Actual  ==================  ================== 


Day# 

Output 

Rain 

ANS 

Climatology 

Daily 

Weekly 

Monthly 

Date 

32 

30% 

0 

0.090 

0.044 

-106.2% 

June  1 

33 

30% 

0 

0.090 

0.044 

-106.2% 

34 

20% 

1 

0.640 

0.626 

-2.3% 

35 

50% 

0 

0.250 

0.044 

-472.9% 

36 

0% 

0 

0.000 

0.044 

100.0% 

37 

0% 

0 

0.000 

0.044 

100.0% 

38 

0% 

0 

0.000 

0.044 

100.0% 

-20.5% 

39 

0% 

0 

0.000 

0.044 

100.0% 

-10.4% 

40 

0% 

0 

0.000 

0.044 

100.0% 

-0.3% 

41 

20% 

0 

0.040 

0.044 

8.3% 

5.1% 

42 

10% 

1 

0.810 

0.626 

-29.4% 

4.2% 

43 

40% 

0 

0.160 

0.044 

-266.6% 

-13.8% 

44 

60% 

1 

0.160 

0.626 

74.4% 

20.4% 

45 

50% 

0 

0.250 

0.044 

-472.9% 

3.4% 

46 

50% 

1 

0.250 

0.626 

60.1% 

18.6% 

47 

70% 

1 

0.090 

0.626 

85.6% 

33.2% 

48 

60% 

1 

0.160 

0.626 

74.4% 

41.6% 

49 

60% 

0 

0.360 

0.044 

-724.9% 

45.7% 

50 

50% 

1 

0.250 

0.626 

60.1% 

52.7% 

51 

60% 

1 

0.160 

0.626 

74.4% 

52.7% 

52 

60% 

1 

0.160 

0.626 

74.4% 

62.4% 

53 

50% 

0 

0.250 

0.044 

-472.9% 

55.5% 

54 

30% 

1 

0.490 

0.626 

21.7% 

43.1% 

55 

30% 

0 

0.090 

0.044 

-106.2% 

33.2% 

56 

50% 

1 

0.250 

0.626 

60.1% 

48.7% 

57 

50% 

1 

0.250 

0.626 

60.1% 

48.7% 

58 

0% 

0 

0.000 

0.044 

100.0% 

43.4% 

59 

0% 

0 

0.000 

0.044 

100.0% 

35.2% 

60 

0% 

0 

0.000 

0.044 

100,0% 

47.4% 

61 

10% 

0 

0.010 

0.044 

77.1% 

59.2% 

36.6% 

June  30 

62 

50% 

1 

0.250 

0.563 

55.6% 

61.8% 

37.8% 

July  1 

63 

70% 

0 

0.490 

0.062 

-687.1% 

29.9% 

32.7% 

64 

60% 

1 

0.160 

0.563 

71.6% 

33.3% 

35.7% 

65 

40% 

1 

0.360 

0.563 

36.1% 

32.6% 

36.3% 

66 

50% 

0 

0.250 

0.062 

-301.6% 

20.1% 

36.7% 

67 

20% 

1 

0.640 

0.563 

-13 . 6% 

10.8% 

36,1% 

68 

40% 

6 

6 1 160 

0 !  062 

-157.0% 

~5 !  3% 

34  I  6% 

69 

20% 

0 

0.040 

0.062 

35.7% 

-8.3% 

34.3% 

70 

20% 

0 

0.040 

0.062 

35.7% 

14.9% 

34.0% 

71 

20% 

0 

0.040 

0.062 

35.7% 

-6.4% 

33.7% 

72 

20% 

0 

0.040 

0.062 

35.7% 

-29.2% 

33.5% 

73 

20% 

0 

0.040 

0.062 

35.7% 

-6.8% 

33.6% 

A-13 


j 


74 

30 

75 

20 

76 

5 

77 

0 

78 

0 

79 

0 

80 

0 

81 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

0 

1 

0 

1 

0 

0 

0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0, 

0 

0. 

0 

0, 

0 

0, 

0 

0, 

0 

0, 

0 

0. 

0 

0. 

0 

0. 

0 

0, 

0 

0, 

0 

0, 

0 

0, 

-44.6% 

-3.3% 

37.2% 

-13.6% 

0.7% 

35.7% 

96.0% 

4.7% 

33.5% 

100.0% 

9.0% 

36.3% 

100.0% 

13.3% 

35.1% 

100.0% 

17.5% 

31.8% 

100.0% 

21.8% 

28.9% 

100.0% 

31.4% 

33.7% 

100.0% 

99.4% 

31.9% 

100.0% 

100.0% 

28.6% 

96.0% 

99.4% 

24.5% 

35.7% 

90.2% 

28.2% 

-13.6% 

27.1% 

25.0% 

36.1% 

27.5% 

26.8% 

-44.6% 

21.2% 

22.6% 

157.0% 

10.1% 

16.2% 

-44.6% 

3.8% 

14.8% 

35.7% 

1.2% 

14.4% 

-13.6% 

-4.2% 

11.1%  July  31 

-72.5% 

9 . 9%  Aua  1 

.052 


58.0% 

6.8% 

10.4% 

39.5% 

18.2% 

19.7% 

39.5% 

27.6% 

17.0% 

379.2% 

20.9% 

11.9% 

39.5% 

24.2% 

17.3% 

206.7% 

27.9% 

18.3% 

-7.5% 

22.8% 

17.6% 

-72.5% 

12.5% 

16.7% 

-72.5% 

2.2% 

15 . 8% 

-72.5% 

-15.7% 

14.9% 

206.7% 

-9.5% 

13.0% 

23.3% 

-39.8% 

12.9% 

23.3% 

-26.6% 

13.5% 

95.2% 

-40.3% 

16.7% 

23.3% 

-26.7% 

15.9% 

23.3% 

-13.0% 

15.1% 

206.7% 

-32.1% 

12.3% 

23.3% 

0.7% 

11.5% 

99.2% 

11.6% 

11,4% 

99.2% 

22.4% 

11.2% 

99.2% 

23.0% 

11.1% 

23.3% 

23.0% 

10.3% 

95.2% 

33.3% 

10.1% 

23.3% 

66.1% 

9.9% 

23.3% 

66.1% 

12.5% 

23.3% 

206.7% 

55.3% 

11.6% 

9.9% 

8.3% 

23.3% 

0.7% 

10.6% 

8.9% 

12.0% 

379.2% 

-58.8% 

7.6% 

LFM/MOS  POP  FORECAST  SKILL  SCORE 
0-12  HRS  POP  FORECASTER  FOR  HARTSFIELD  INTERNATIONAL  AIRPORT 


June- July-August  Skill  Score  -  16.6% 


DaY* 


Squared-Errors  Skill  Scores 

Output  Rain  ANS  Climatology  Daily  Weekly  Monthly  Date 


32 

20% 

0 

0.040 

0.044 

33 

30% 

0 

0.090 

0.044 

34 

20% 

1 

0.640 

0.626 

35 

30% 

0 

0.090 

0.044 

36 

0% 

0 

0.000 

0.044 

37 

0% 

0 

0.000 

0.044 

38 

0% 

0 

0.000 

0.044 

39 

0% 

0 

0.000 

0.044 

40 

0% 

0 

0.000 

0.044 

41 

2% 

0 

0.000 

0.044 

42 

20% 

1 

0.640 

0.626 

43 

30% 

0 

0.090 

0.044 

44 

60% 

1 

0.160 

0.626 

45 

60% 

0 

0.360 

0.044 

46 

30% 

1 

0.490 

0.626 

47 

60% 

1 

0.160 

0.626 

48 

40% 

1 

0.360 

0.626 

49 

50% 

0 

0.250 

0.044 

50 

50% 

1 

0.250 

0.626 

51 

50% 

1 

0.250 

0.626 

52 

30% 

1 

0.490 

0.626 

53 

50% 

0 

0.250 

0.044 

54 

20% 

1 

0.640 

0.626 

55 

20% 

n 

V 

0.040 

0.044 

56 

40% 

1 

0.360 

0.626 

57 

40% 

1 

0.360 

0.626 

58 

0% 

0 

0.000 

0.044 

59 

0% 

0 

0.000 

0.044 

60 

0% 

0 

0.000 

0.044 

61 

20% 

0 

0.040 

0.044 

62 

40% 

1 

0.360 

0.563 

63 

60% 

0 

0.360 

0.062 

64 

50% 

1 

0.250 

0.563 

65 

40% 

1 

0.360 

0.563 

66 

40% 

0 

0.160 

0.062 

67 

20% 

1 

0.640 

0.563 

68 

20% 

0 

0.040 

0.062 

69 

20% 

0 

0.040 

0.062 

70 

20% 

0 

0.040 

0.062 

71 

20% 

0 

0.040 

0.062 

72 

20% 

0 

0.040 

0.062 

73 

20% 

0 

0.040 

0.062 

8.3%  June  1 

-106.2% 

-2.3% 

-106.2% 

100.0% 

100.0% 

100.0%  3.1% 

100.0%  7.6% 

100.0%  17.8% 

99.1%  70.4% 

-2.3%  27.9% 

-106.2%  17.7% 

74.4%  39.4% 

-724.9%  14.9% 

21.7%  15.2% 

74.4%  27.9% 

42.5%  29.7% 

-472.9%  29.0% 

60.1%  36.9% 

60.1%  34.1% 

21.7%  40.8% 

-472.9%  37.5% 

-2.3%  22.6% 

8.3%  17.6% 

42.5%  29.1% 

42.5%  25.7% 

100.0%  18.8% 

100.0%  19.6% 

100.0%  31.8% 

8.3%  45.6%  27.1%  June  30 

36.1%  43.7%  27.6%  July  1 

-478.3%  21.5%  24.1% 

55.6%  25.9%  26.1% 

36.1%  27.2%  27.2% 

-157.0%  19.5%  27.9% 

-13.6%  10.4%  26.2% 

35.7%  11.1%  25.9% 

35.7%  4.6%  25.7% 

35.7%  21.1%  25.4% 

35.7%  8.2%  25.1% 

35.7%  -6.8%  24.9% 

35.7%  6.1%  24.6% 


A- 15 


74 

20% 

0 

0.040 

0.062 

35.7% 

35.7% 

26.5% 

75 

10% 

1 

0.810 

0.563 

-43.8% 

-12.1% 

23.1% 

76 

10% 

0 

0.010 

0.062 

83.9% 

-8.9% 

20.1% 

77 

10% 

0 

0.010 

0.062 

83.9% 

-5.7% 

23.9% 

78 

10% 

0 

0.010 

0.062 

83.9% 

-2.5% 

24.5% 

79 

5% 

0 

0.003 

0.062 

96.0% 

1.5% 

21.3% 

80 

0% 

0 

0.000 

0.0  62 

100.0% 

5.8% 

20.2% 

81 

0% 

0 

0.000 

0.062 

100.0% 

10.1% 

23.6% 

82 

2% 

0 

0.000 

0.062 

99.4% 

92.4% 

21.1% 

83 

0% 

0 

0.000 

0.062 

100.0% 

94.7% 

18.2% 

84 

20% 

0 

0.040 

0.062 

35.7% 

87.9% 

18.0% 

85 

20% 

0 

0.040 

0.062 

35.7% 

81.0% 

21.7% 

86 

20% 

1 

0.640 

0.563 

-13.6% 

23.1% 

20.9% 

87 

30% 

1 

0.490 

0.563 

13.0% 

15.8% 

20.3% 

88 

20% 

0 

0.040 

0.062 

35.7% 

13.0% 

18.1% 

89 

20% 

0 

0.040 

0.062 

35.7% 

10.3% 

15.5% 

90 

20% 

0 

0.040 

0.062 

35.7% 

7.5% 

15.1% 

91 

20% 

0 

0.040 

0.062 

35.7% 

7.5% 

14.6% 

92 

20% 

1 

0.640 

0.563 

-13.6% 

0.5% 

11.3% 

July  31 

93 

30% 

0 

0.090 

0.052 

-72.5% 

3.3% 

10.6% 

Aug  1 

94 

30% 

1 

0.490 

0.595 

17.7% 

5.5% 

9.0% 

95 

40% 

1 

0.360 

0.595 

39.5% 

14.7% 

16.4% 

96 

30% 

1 

0.490 

0.595 

17.7% 

14.9% 

13.1% 

97 

50% 

0 

0.250 

0.052 

-379.2% 

6.2% 

7.6% 

98 

40% 

1 

0.360 

0.595 

39.5% 

12.1% 

12.0% 

99 

30% 

0 

0.090 

0.052 

-72.5% 

16.1% 

13.7% 

100 

20% 

1 

0.640 

0.595 

-7.5% 

13.0% 

11.6% 

101 

30% 

0 

0.090 

0.052 

-72.5% 

10.2% 

10.7% 

102 

20% 

0 

0.040 

0.052 

23.3% 

1.7% 

10.6% 

103 

30% 

0 

0.090 

0.052 

-72.5% 

-7.5% 

9.7% 

104 

40% 

0 

0.160 

0.052 

-206.7% 

-1.3% 

7.7% 

105 

20% 

0 

0.040 

0.052 

23.3% 

-26.6% 

7.6% 

106 

10% 

0 

0.010 

0.052 

80.8% 

-17.8% 

7.9% 

107 

20% 

0 

0.040 

0.052 

23.3% 

-28.7% 

12.8% 

108 

20% 

0 

0.040 

0.052 

23.3% 

-15.0% 

12.2% 

109 

20% 

0 

0.040 

0.052 

23.3% 

-15.0% 

11.5% 

110 

20% 

0 

0.040 

0.052 

23.3% 

-1.3% 

10.9% 

111 

20% 

0 

0.040 

0.052 

23.3% 

31.5% 

10.1% 

112 

2% 

0 

0.000 

0.052 

99.2% 

42.4% 

10.0% 

113 

2% 

0 

0.000 

^.052 

99.2% 

45.0% 

9.8% 

114 

2% 

0 

0.000 

0.052 

99.2% 

55.9% 

9.7% 

115 

20% 

0 

0.040 

0.052 

23.3% 

55.9% 

8.8% 

116 

20% 

0 

0.040 

0.052 

23.3% 

55.9% 

8.7% 

117 

30% 

0 

0.090 

0.052 

-72.5% 

42.2% 

7.7% 

118 

20% 

0 

0.040 

0.052 

23.3% 

42.2% 

10.0% 

119 

20% 

0 

0.040 

0.052 

23.3% 

31.3% 

9.8% 

120 

30% 

0 

0.090 

0.052 

-72.5% 

6.8% 

8.6% 

121 

30% 

0 

Q.  090 

0.052 

-72.5% 

-17.8% 

7.4% 

122 

30% 

0 

0.090 

0.052 

-72.5% 

-31.4% 

6.2% 

123 

50% 

0 

0.250 

0.052 

-379.2% 

-89.0% 

1.7% 

Aug  31 
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2.032360 

AA100V 

GOPAD  103 

2.031388 

CA900V 
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WA500T 

138 
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1.903888 

AP700D 
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ABSTRACT 

The  wid-'  dispersion  of  Army  forces  on  the  modem  battlefield  and  the 
complexity  of  current  weapons  systems  have  increased  the  need  for  forecasted 
weather  data  that  could  be  used  as  input  to  Tactical  Decision  Aids.  The 
Integrated  Meteorological  System  (IMETS)  will  enable  the  Staff  Weather 
Officer  (SWO)  to  support  Division  and  Corp  staffs  who  are  primarily  engaged 
in  planning.  These  staffs  and  other  operational  units  need  a  diverse  set  of 
weather  parameters  on-demand  for  specific  locations  and  specific  time 
frames.  Research  and  development  work  investigated  the  potential  of  a 
software  program  called  Goal  Oriented  Pattern  Detection  (GOPAD)  to 
produce  tactical  weather  forecasting  models.  This  paper  describes:  (1)  how 
GOPAD  operates,  (2)  the  types  of  tactical  mesoscale  forecast  models  that 
could  be  developed,  (3)  a  method  for  defining  forecast  models,  and  (4)  the 
GOPAD  development  process  used  to  create  new  forecast  models. 

1.  THE  ROLE  OF  AUTOMATED  WEATHER 
FORECASTING  ON  THE  BATTLEFIELD 

The  wide  dispersion  of  Army  forces  on  the  modem  battlefield  and  the  complexity  of 
current  weapons  systems  have  increased  the  need  for  forecasted  weather  data  that  could  be 
used  as  input  to  Tactical  Decision  Aids  (TDA)  that  perform  weather  effects  analysis. 

Weather  conditions  affect  various  types  of  military  units  in  different  ways,  depending  upon 
the  type  of  unit,  the  mission,  the  enemy  situation,  and  the  terrain.  Since  weather  varies  with 
time  and  locality,  weather  forecasting  and  effects  analysis  must  be  constantly  reappraised  to 
retain  its  usefulness  as  combat  intelligence.  The  complexity  of  this  analysis  process,  the 
premium  placed  on  timeliness,  and  the  availability  of  tactical  computers  requires  that  the 
forecasting,  analysis,  and  display  techniques  be  automated.  Consequently,  research  and 
development  is  ongoing  to  devise  new  ways  of  automating  mesoscale  weather  parameter 
forecasting  in  the  battlefield  environment. 

We  believe  that  weather  parameter  forecasting  is  one  of  the  fundamental  functional 
requirements  for  the  Integrated  Meteorological  System  (IMETS)  because  EMETS  will  support 
Division  and  Corps  staffs  who  are  primarily  engaged  in  planning.  These  staffs  need  a  diverse 
set  of  forecasted  weather  parameters  on-demand  for  specific  locations  and  times  frames.  The 
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incorporation  of  automated  weather  forecasting  models  in  IMETS  is  therefore  crucial  to  the 
Staff  Weather  Officer's  (SWO)  ability  to  provide  forecasted  weather  parameters  for  input  to  a 
myriad  of  Tactical  Decision  Aids  (TDA). 

The  Required  Operational  Capability1  (ROC)  states  that  IMETS  will  provide  the 
commander  with  weather  forecasts  in  the  operational  area.  These  documents  reflect  the 
critical  importance  of  automated  weather  parameter  forecasting  to  IMETS.  Clearly,  IMETS 
must  enable  the  SWO  to  supervise,  control,  understand,  and  subsequently  brief  the  current 
and  projected  weather  to  the  commander.  But  EMETS  must  also  enable  the  SWO  to  provide 
forecasts  for  a  diverse  set  of  weather  parameters  that  will  be  used  as  input  to  many  TDAs. 
The  SWO  cannot  be  expected  to  manually  produce  individual  forecasts  for  all  the  weather 
parameters  required  by  all  Corps/division/Brigade  staffs  and  operational  units,  as  needed,  for 
specific  locations  and  time  frames. 

In  addition,  the  SWO  and  his  staff  may  not  be  very  familiar  with  local  conditions  for 
the  area  of  interest.  Consequently,  the  forecasting  performance  of  humans  in  a  tactical 
situation  might  be  highly  variable  when  positive  skill  and  consistent  performance  are  crucial. 
Thus,  automated  weather  forecast  models  are  essential  to  a  SWO-based  forecasting  process 
and  should  be  considered  as  yet  another  type  of  decision  aid.  For  example,  the  SWO  would 
use  automated  forecast  models  just  as  the  National  Weather  Service  forecasters  use  the 
Limited  Fine  Mesh/Model  Output  Statistics  (LFMXMOS)  models. 

.  Assuming  that  there  is  a  critical  need  for  a  highly  automated  process  which  is  under 
the  control  of  the  SWO,  then  the  next  question  is  how  do  we  produce  these  weather  forecast 
models?  Should  these  models  be  based  upon  statistical  modeling  techniques,  expert  system, 
neural  nets,  chaos  theory,  satellite-based  image  understanding,  cr  some  appropriate 
combination?  What  are  the  expected  time  and  resource  costs  to  produce  these  weather 
forecast  models  using  any  approach  verses  the  potential  performance  of  each  approach?  The 
answers  to  these  types  of  questions  is  one  of  the  goals  of  our  research. 

2.  OVERVIEW  OF  GOPAD 


CCI  is  investigating  the  potential  of  a  neural  net  approach  to  produce  tactical 
weather  forecast  models.  This  neural  net  research  is  based  upon  a  proprietary  algorithm, 
called  Goal  Oriented  Pattern  Detection  (GOPAD),  developed  by  Dr.  Kenneth  Young  at 
the  Institute  of  Atmospheric  Physics,  University  of  Arizona.  GOPAD  has  been  used  to 
produce  two  research  models-GOPAD-Atlanta-RIR  and  GOPAD  RT-89.  The  GOPAD- 
Atlanta-RIR  was  ?  Probability  of  Precipitation  ToP)  model  developed  for  Hartsfield 
International  Airport  in  Atlanta,  Georgia.  The  GOPAD  RT-89  was  a  severe  and  significant 
weather  forecast  model  develop  for  the  SHOOTOUT-89  exercise  sponsored  by  NOAA/FSL  in 
Boulder,  Colorado. 


In  general,  GOPAD  first  identifies  the  optiro  statistical  relationships  between  the 
variables  and  indices  from  an  arbitrarily  large,  correlated  weather  data  set,  and  then 
develops  a  non-linear  forecast  model.  This  model  development  process  is  also  able  to  reveal 
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exemplars.  GOPAD  extends  the  multi-discriminant  analysis  (MDA)  methods  developed  by 
Miller  (1962)  and  the  analogue  forecasting  method  of  Kruizinga  and  Murphy  (1983).  The 
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GOPAD  approach  is  a  k-nearest  neighbor  search  which  is  similar  to  Leon  Cooper's  Keduced 
Coulomb  Energy  (RCE)  algorithm. 

The  GOPAD  model  development  software  is  actually  a  set  of  three  separate  programs 
called  FUZAN,  FUZPICK,  and  FUZUP.  The  first  two  programs  are  the  statistical  front-end 
to  the  third,  which  is  a  neural  net  type  of  program. 

The  first  program  (FUZAN)  analyzes  all  the  candidate  predictor  variables  one  at  a 
time  to  determine  their  individual  potential  contribution  of  information  to  the  forecast  model 
to  be  developed.  Once  this  is  completed,  it  creates  linear  combinations  of  the  variables  that 
are  considered  by  the  developer  to  be  highly  correlated  (e.g.,  all  temperature  variables,  etc.). 
FUZAN  systematically  creates  increasingly  complex  indices  from  all  the  specified  variables 
until  the  Chi-square  value  cannot  be  increased  by  a  more  complex  index.  The  creation  of 
indices  from  highly  correlated  variables  serves  to  improve  the  signal-to-noise  ratio  and 
sharpen  the  overall  forecast  ability  of  the  model. 


In  the  GOPAD  RT-89  model  there  were  approximately  1400  candidate  rawinsonde 
and  mesonet  predictor  variables  for  three  years- 1983,  1985,  and  1987.  FUZAN  used  these 
variables  to  automatically  create  about  120  candidate  predictor  indices.  FUZAN  then 
analyzed  all  1520  candidate  variables  and  indices  to  choose  a  subset  of  400-500  variables  and 
indices  for  further  processing  by  the  second  program. 

The  second  program  called  FUZPICK  selects  the  optimal  combination  of  variables 
and  indices  that  contain  virtually  all  the  predictive  information  in  the  original  data  set.  In 
the  GOPAD  RT-89  six  sub-models,  there  were  3  to  6  variables/indices  that  were  ultimately 
selected.  All  together  there  are  about  40  individual  predictor  variables  that  are  required  to 
run  the  GOPAD  RT-89  model  (indices  may  combine  up  to  10  individual  predictor  variables). 

The  third  program  called  FUZUP  optimizes  the  n-space  scaling  and  the  neighborhood 
size  to  maximize  a  measure  of  forecast  performance  (e.g.,  Brier-based  skill  score).  One  major 
advantage  of  this  process  is  that  it  allows  direct  access  to  all  the  other  forecast  parameters 
that  were  not  used  in  training  the  model.  This  enables  a  GOPAD  forecast  model  to  display  a 
wide  variety  of  forecast  parameters  derived  from  the  data  variables. 

The  forecast  model  is  based  on  the  output  of  FUZUP  which  includes  the  historical 
data  base  of  the  predictor  variables  chosen  by  FUZPICK  Forecasts  are  made  using  analogs 
selected  from  the  historical  data  set.  The  number  of  analogs  is  the  neighbor  size  as  optimized 
by  FUZUP.  The  forecast  model  can  also  be  used  to  generate  exemplars  to  illustrate  the 
patterns  leading  to  a  particular  forecast. 


There  are  three  versions  of  the  GOPAD  software-GOPAD  I,  GOPAD  II,  and  GOPAD 
III.  GOPAD  I  operates  on  an  IBM-compatible  8386  type  computer.  GOPAD  I  was  used  to 
produce  the  GOPAD-Atlanta  Probability  of  Precipitation  model.  GOPAD  II  operates  on  a 
VAX  3200.  GOPAD  II  was  used  to  produce  the  GOPAD  RT-89  severe/significant  weather 
forecasting  model.  GOPAD  III  (presently  in  development)  operates  on  a  VAX  3200  which  is  a 
host  for  Interstate  Electronics'  Quen  supermini-computer  rated  at  150  Megaflops.  GOPAD 
III 
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of  raw  weather  parameters  and  larger  data  bases. 
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3.  TACTICAL  MESOSCALE  FORECAST  MODELS 


The  GO  PAD  development  software  can  be  used  to  automate  the  creation  of  a  wide 
variety  of  tactical  forecast  models.  Four  of  the  most  difficult  tactical  mesoscale  forecasting 
problems  are  (1)  fog  as  it  is  related  to  visibility  and  ceiling;  (2)  precipitation  amounts  and 
type;  (3)  severe  weather  such  as  tornadoes,  hail,  lightning,  and  wind  gusts;  and  (4)  cloud 
distributions,  types,  and  coverage.  The  following  paragraphs  describe  the  type  of  information 
that  a  GO  PAD  model  for  each  of  these  forecasting  problems  could  provide. 

A  GO  PAD  based  fog  model  would  forecast  formation  and  dissipation  times,  and 
graphically  display  visibility  and  ceiling.  The  operator  would  be  able  to  specify  the 
probability  that  visibility  or  ceilings  would  be  below  a  specified  value  on  a  time  axis.  The 
operator  would  be  able  to  produce  maps  showing  regions  of  visibility  or  ceiling  using  a  zoom 
capability.  The  user  would  be  able  to  display  a  wide  variety  of  exemplar  conditions  to  help 
explain,  confirm,  modify,  or  reject  a  computer-generated  forecast  Fog  models  might  use 
terrain  information,  the  output  from  a  model  of  wind  flow  over  complex  terrain,  local 
rawinsonde  data,  low-level  cloud  information  from  a  satellite,  and  output  from  a  synoptic 
scale  forecast  model.  Ground  truth  would  be  derived  from  hourly  surface  observations  firom  a 
variety  of  locations. 

A  GO  PAD  based  precipitation  model  would  display  isopleths  of  expected  amounts 
and  types  (e.g.,  rain,  snow,  freezing  rain,  etc.).  The  user  would  be  able  to  selectively  display 
the  probability  of  precipitation  greater  than  a  specified  amount  in  a  specified  period.  A  user 
would  be  able  to  display  the  expected  total  cumulative  amount  of  precipitation  expected 
within  a  specified  period.  The  user  would  be  able  to  predict  rain  intensities  as  a  function  of 
time  and  within  a  specified  confidence  interval.  The  user  would  be  able  to  display  a  wide 
variety  of  exemplar  conditions  to  help  explain,  confirm,  modify,  or  reject  a  computer¬ 
generated  forecast.  These  models  might  use  the  output  from  a  synoptic  scale  numerical 
model,  local  rawinsonde  data,  and  satellite-derived  cloud  track  winds,  total  ozone 
measurements,  precipitable  water,  etc.  Ground  truth  would  be  obtained  from  archived 
records. 


A  GO  PAD  based  severe/significant  weather  model  would  forecast  the 
probabilities  of  occurrence  of  severe  weather  events  such  as  tornadoes,  hail,  lightning,  and 
high  winds  as  a  function  of  time.  However,  spatial  resolution  of  severe  weather  phenomena 
greater  than  one  hour  in  advance  on  a  scale  of  ten  kilometers  or  less  is  unlikely.  The  user 
would  be  able  to  display  a  wide  variety  of  exemplar  conditions  to  help  explain,  confirm, 
modify,  or  reject  a  computer-generated  forecast  These  models  might  use  the  output  from  a 
synoptic  scale  numerical  model,  local  rawinsonde  data,  conventional  and  Doppler  radar  data, 
wind  profiler  data,  surface  meso-net  data,  mesoscale  models,  and  satellite-derived  cloud  track 
winds,  total  ozone  measurements,  precipitable  water,  etc.  Ground  truth  data  would  be 
obtained  from  conventional  surface  observations,  meso-net  data,  and  severe  weather  studies 
like  the  real-time  exercises  conducted  every  two  years  by  NOAA/ERL  in  northeast  Colorado. 


A  GOPAD  based  cloud  model  would  forecast  a  three-dimensional  distribution  of 
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indication  of  basic  cloud  type  (e.g.,  cumuloform  or  stratiform).  The  display  would  use  this 
fraction  to  randomly  distribute  clouds  within  each  volume.  Typical  cloud  sizes  would  be 
estimated  from  the  forecasted  cloud  type  and  climatology.  Sophisticated  display  techniques 
would  enable  the  user  to  specify  an  arbitrary  point  and  direction  of  sight,  and  display  a  cloud- 
free  line-of-sight  view.  The  capability  would  also  be  applied  to  non-visible  wavelengths  such 
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as  infrared.  Consequently,  forecast  models  would  also  be  developed  to  predict  parameters 
like  temperature  and  water  vapor  profiles  that  afreet  non-visual  wavelength  visibilities.  The 
user  would  be  able  to  display  a  wide  variety  of  exemplar  conditions  to  help  explain,  confirm, 
modify,  or  reject  a  computer-generated  forecast  These  models  might  use  the  output  from  a 
synoptic  scale  numerical  model,  satellite-derived  cloud  track  winds,  cloud  types,  heights,  and 
amounts  derived  from  satellites,  and  local  rawinsonde  data.  Ground  truth  would  be  obtained 
from  surface  observations  and  satellites. 

4.  WEATHER  FORECAST  MODEL  DEFINITION 

In  order  to  completely  describe  a  forecast  model  at  least  four  components  should 
be  specified:  (1)  the  meteorological  event  to  be  forecasted;  (2)  the  sources  of  historical  input 
variables  and  ground  truth;  (3)  the  lead  time  and  forecast  period;  (4)  the  size  of  the  forecast 
region.  Each  of  these  four  components  are  described  in  the  following  paragraphs. 

4.1  THE  EVENT  TO  BE  FORECASTED 

Precipitation 

(a)  PoP  by  type  (rain,  snow,  freezing  rain) 

(b)  estimate  of  precipitation  amount  with  confidence  limits 

(c)  probability  of  precipitation  (PoP)  greater  than  specified  amount 

Temperature 

(a)  expected  temperature  with  confidence  limits 

(b)  probability  of  temperature  above  or  below  specified  value 

"  WINDS 

(a)  expected  wind  direction,  speed,  and  gusts  with  confidence  limits 

(b)  probability  of  winds  above  or  below  specified  speed 

Cloud  Cover,  Ceiling,  and  Visibility 

(a)  expected  cloud  cover,  ceiling,  and  visibility  with  confidence  limits 

(b)  probability  of  an  event  above  or  below  specified  values 

(c)  expected  cloud  types  and  coverage  with  confidence  limits 

Severe  Weather 

(a)  probabilities  of  funnel  cloud  or  tornado 

(b)  probabilities  of  hail  larger  than  specified  sizes 

(c)  probability  of  wind  gusts  greater  than  specified  speed 

(d)  probability  of  cloud-to-ground  lightning  within  specified  grids 

Although  there  are  many  environmental  events  that  can  be  predicted,  the  GOPAD 
model  development  process  requires  a  fairly  accurate  training  set  (i.e.,  a  historical  data  base) 
in  order  to  develop  a  model  to  forecast  the  desired  event.  We  have  found  that  as  a  minimum 
these  data  bases  should  include  about  100  data  points  for  climate  forecasters  or  about  500 
data  points  for  shorter  term  forecasters. 

Another  important  criteria  for  determining  how  well  an  event  can  be  forecasted  is 
that  there  must  be  a  minimum  number  of  occurrences  of  that  event  in  the  historical  data 
base.  For  example,  a  rare  event  (e.g.,  a  tornado)  m,.  y  occur  with  a  frequency  of  0.1%.  A  data 
set  containing  500  data  points  would  contain  nr-'y  5  tnmado  events.  We  find  that  at  least  10 
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or  more  occurrences  are  required  to  accurately  forecast  a  rare  event  If  the  historical  data  set 
does  not  contain  a  sufficient  number  of  occurrences  of  an  event,  then  more  data  and/or 
"multiplexing”  may  be  required. 

In  order  to  achieve  the  optimal  level  of  performance  given  a  limited  number  of  ground 
truth  events  in  the  training  set  we  experimented  with  two  techniques  to  mitigate  this 
problem  which  we  refer  to  as  the  "rare  event”  problem.  First,  a  new  technique,  called 
multiplexing  (plexing),  was  developed  to  enable  different  sets  of  ground  truth  events  to  be 
used  in  the  creation  of  a  model  thereby  increasing  the  number  of  occurrences  of  the  event  in 
the  training  set.  The  concept  of  multiplexing  was  tested  on  a  very  small  scale  using 
geographically  similar  contiguous  regions  during  SHOOTOUT-89.  GO  PAD  RT-89  used 
multiplexing  to  extend  a  historical  training  set  that  contained  only  three  years  of  ground 
truth  data  for  forecasting  severe  storm  parameters  and  two  years  of  ground  truth  data  for 
forecasting  significant  storm  parameters.  GOPAD  RT-89  is  composed  of  six  submodels.  One 
submodel  is  single-plexed,  four  are  two-plexed,  and  one  is  three-plexed.  By  comparison,  a 
site-specific  scenario  model  might  require  as  many  as  20-plexes.  Methods  for  selecting  the 
optimal  number  of  plexes  for  a  given  problem  have  been  addressed  only  in  a  preliminary 
manner.  It  is  also  fair  to  note  that  greater  plexing  will  be  more  computationally  expensive. 

Second,  a  self-adapting  forecast  model  that  learns  as  it  is  being  used  was  also 
developed  for  SHOOTOUT-89  to  evaluate  the  potential  of  this  technique  for  mitigating  the 
lack  of  ground  truth  data.  We  are  presently  evaluating  the  skill  of  our  self-adapting  model 


compared  to  an  identical  but  static  model  to  estimate  the 

potential  learning  curve. 

4.2.  SOURCES  OF  PREDICTOR  VARIABLES 

A 

Mesonet  or  other  surface  data 

F. 

Satellite  derived  cloud  track  winds 

B. 

Rawinsonde  data 

G. 

Other  satellite  derived  variables 

C. 

Barotropic  model  output 

H. 

Topographic  or  terrain  data 

D. 

Nested  Grid  Model  (NGM)  output 

I. 

Worldwide  monthly  average 

E. 

Mesoscale  primitive  equation 

temperatures 

model  output 

J. 

Worldwide  monthly  average  sea 
level  pressures 

This  list  shows  many,  but  not  all,  of  the  possible  sources  of  predictor  variables  that 
could  be  used  to  construct  a  model.  The  selection  of  the  variables  to  use  depends  upon  many 
factors.  One  of  the  major  cost  factors  in  developing  a  model  is  the  number  of  new  historical 
data  bases  that  must  be  developed  for  a  particular  site  or  region.  Often  these  historical  data 
bases  can  be  formed  one  time  for  very  large  regions  (e.g.,  several  states  in  size)  and, 
subsequently,  be  used  to  produce  a  variety  of  models  in  the  same  region  or  in  different  areas 
of  the  same  region.  After  the  data  bases  are  developed  and  organized  for  processing,  GOPAD 
model  construction  is  primarily  man-out-of-the-loop. 

Generally,  the  best  models  will  be  produced  when  the  widest  possible  variety  of  input 
variables  are  used  to  develop  a  forecast  model.  This  variety  should  include  variables  that 
may  never  have  been  thought  to  be  important  enough  to  consider  in  a  model.  The  number  of 
individual  candidate  variables  that  can  be  included  in  a  GOPAD  model  could  number  from 
100  to  8,000  or  more.  It  also  makes  no  difference  whether  these  variables  are  highly 
correlated  or  not.  The  GOPAD  model  development  process  creates  a  near  optimal  model  for  a 
given  data  set,  no  matter  how  much  redundancy  is  present  in  the  training  set.  Normally,  the 
redundant  variables  are  automatically  combined  by  GOPAD  to  form  optimal  machine-derived 
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indices  which  usually  contain  much  more  predictive  information  than  any  of  the  individual 
candidate  variables  from  which  they  were  derived.  We  find  that  these  indices  are  often  so 
complex  that  they  cannot  be  easily  explained.  Nevertheless,  the  indices  that  are  selected 
have  extremely  high  Chi-square  values  (e.g.,  over  100)  indicating  very  high  predictive  power. 

The  only  limitation  pertaining  to  these  input  variables  is  that  they  should  be 
continuous,  rather  than  binary  or  discrete,  with  few  possible  values.  For  example,  discrete 
variables  with  relatively  few  values  (i.e.,  less  than  10)  do  not  facilitate  optimization  of  the 
separation  between  these  multi-dimensional  data  points.  In  the  weather  domain,  the  raw 
variables  are  generally  continuous  variables,  so  this  limitation  is  usually  not  important. 

Finally,  the  size  of  the  training  set  should  be  as  large  as  possible.  For  example, 
these  data  base  should  contain  1,000  or  more  data  points.  Consequently,  the  size  of  these 
data  bases  are  usually  on  the  order  of  10-500  megabytes  in  size.  For  example,  the  NGM 
historical  data  base  for  just  the  past  four  years  is  about  5-7  gigabytes  in  size,  but  only  about 
500  MB  of  candidate  predictor  variables  will  be  extracted  for  processing. 

4.3.  FORECAST  LEAD  TIMES  AND  VALID  PERIODS 

FORECAST  TYPES  LEADTIMES  MINIMUM  VALID  PERIODS 


Immediate-range 

0  to  4 

hours 

1 

hour 

Short-range 

6  to  24 

hours 

3 

hours 

Mid-range 

24  to  96 

hours 

12 

hours 

Long-range 

4  to  9 

days 

1 

day 

Extended-range 

10  to  30 

days 

5 

days 

Short-climate 

1  to  3 

months 

10 

days 

Mid-climate 

3  to  9 

months 

30 

days 

Long-climate 

9  to  24 

months 

3 

months 

The  lead  time  is  the  amount  of  lag  between  the  time  that  the  forecast  model 
produces  a  forecast  and  the  time  that  the  valid  period  begins.  The  valid  period  is  the 
timeframe  within  which  the  forecast  is  valid.  For  example,  an  Immediate-range  forecast 
model  may  produce  a  forecast  at  12Z  which  is  valid  for  15Z-16Z.  In  this  example,  the  lead 
time  is  three  hours  and  the  valid  period  is  one  hour.  The  valid  period  may  be  equal  to  or 
longer  than  the  temporal  resolution  of  the  ground  truth  data  used  to  train  the  model.  Valid 
periods  shorter  than  those  listed  are  not  recommended,  but  valid  periods  may  be  longer  than 
those  listed. 

A  forecast  model  with  a  short  valid  period  could  be  very  important  to  the  battlefield 
planning  process.  If  a  series  of  hourly  forecast  models  were  run  at  the  same  time  and  if  their 
forecasts  were  graphically  displayed  on  the  same  time  line,  then  one  could  see  the  temporal 
evolution  of  the  weather  event  being  forecasted.  We  refer  to  this  type  of  combined  forecast 
model  as  a  scenario  model.  A  scenario  model  will  make  it  possible  for  one  to  see  breaks  or 
windows  in  the  weather.  In  addition,  a  scenario  model  would  support  the  tactical  planning 
and  decision-making  process  at  Division  and  Corps  hv  making  forecast  information  readily 
available  for  any  combination  of  lead  time  and  valid  period  that  might  be  required  by  a 
myriad  of  the  Tactical  Decision  Aids  (TDA). 
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4.4.  SIZE  OP  FORECAST  REGION 


Fixed  Point  Models 

(a)  Single-Site  Model-one  model  forecasts  for  one  site. 

(b)  Multi-Site  Model-one  model  forecasts  for  several  sites. 

Regional  Models 

(a)  Low-Res  Single  Region  Model-one  model  provides  a  forecast  to  cover  a  single 
geographical  region. 

(b)  Multi-Region  Model-one  model  forecasts  in  several  regions  that  may  be 
meteorologically  very  similar  and/or  relatively  close  geographically. 

FULL  RESOLUTION  MODELS 

(a)  Hi-Res  Single  Region  Model-one  model  forecasts  for  any/all  specified  points 
within  the  given  geographical  region. 

(b)  Generic  Model-one  model  forecasts  for  any/all  specified  points  in  a  variety  of 
meteorologically  or  geographically  similar  regions. 

Fixed  point  forecast  models  are  trained  on  the  ground  truth  information  that  is 
specific  to  one  or  more  sites.  A  probability  of  precipitation  (PoP)  model  trained  on  the 
amount  of  rain  received  in  one  rain  gauge  is  a  good  example  of  a  single-site  model.  A 
multi-site  model  would  be  trained  on  the  data  from  several  locations  and  therefore  could  be 
used  to  make  predictions  for  any  of  those  sites.  In  either  case,  a  point  model  generally  will 
provide  useful  forecasts  only  for  the  specific  locations  where  it  was  trained. 

A  low-res  single  region  model  will  make  predictions  for  a  region  rather  than  for 
one  or  more  sites  within  a  region.  The  resolution  of  the  ground  truth  data  is  the  region  as  a 
whole.  Consequently,  the  output  of  a  regional  model  only  indicates  the  probabilities  that  a 
specific  event  will  occur  within  the  region  but  will  not  localize  the  event  within  the  region. 

A  multi-region  model  is  a  concept  designed  to  reduce  the  cost  of  creating  many 
regional  models  when  these  areas  are  meteorologically  similar  and  relatively  close 
geographically.  This  concept  suggests  that  when  models  are  required  for  large  regions,  it  is 
possible  to  multiplex  the  ground  truth  data  from  several  smaller  portions  of  the  larger  region 
to  create  a  multi-region  model.  For  example,  the  ground  truth  data  base  for  a  mesoscale 
weather  forecasting  model,  developed  for  four  regions  around  Boulder,  Colorado,  contained  a 
record  of  whether  or  not  severe/significant  weather  had  occurred  anywhere  within  the  region. 

The  hi-res  single  region  model  combines  the  advantage  of  the  point  and  regional 
models  to  provide  point  forecasts  for  any  or  all  of  the  points  in  the  region,  whether  or  not 
there  is  ground  truth  data  available  for  the  desired  region.  In  order  to  make  this  model 
useful,  non-meteorological  information  (e.g.,  terrain  elevation  data,  direction  of  slope, 
vegetation  type,  surface  roughness,  lat-long,  distance  to  ocean,  etc.)  must  be  used  in  addition 
to  the  meteorological  data. 

The  hi-res  generic  model  takes  the  high  resolution  model  one  step  farther.  A 
generic  model,  in  concept,  would  be  portable  to  geographic  locations  where  insufficient 
historical  data  was  available  to  construct  a  model.  This  concept  suggests  that  it  might  be 
possible  to  create  a  model  using  the  historical  data  from  a  variety  of  geographically-different, 
but  meteorologically-similar  locations.  Thus,  a  region-specific  model  for  an  overseas  area 
might  be  initially  developed  for  a  similar  region  in  the  United  States  for  example,  and  once 
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constructed  it  would  be  used  in  the  overseas  location.  Such  a  model  would  be  designed  to 
self-adapt  using  real-time  data  and  post-forecast  ground  truth  information.  Eventually  it 
would  transform  itself  into  a  site-specific  forecast  model  with  a  greater  level  of  skill  than 
when  it  was  first  placed  into  service.  A  generic  model  could  enclose  a  region  of  almost  any 
size  (e.g.,  the  State  of  Georgia,  or  Georgia  and  Alabama)  and  it  would  also  be  trained  with 
meteorological  data  and  non-meteorological  data.  It  is  appropriate  to  note  that  the 
development  of  a  generic  model  will  require  exploration  and  the  development  of  new 
production  techniques. 

Finally,  all  of  these  high  resolution  regional  models  would  isopleth  their  forecasts 
(e.g.,  expected  amount  of  precipitation)  over  the  region.  A  zoom  capability  would  enable 
regions  of  interest  to  be  isoplethed  at  any  desired  resolution.  Similar  isopleth  maps  could  be 
constructed,  for  example,  for  the  minimum  expected  amounts  of  precipitation  with  any  user- 
specified  confidence  limit.  In  this  way,  isopleths  maps  could  also  show  the  probability  that 
precipitation  would  exceed  any  user-specified  amount.  The  list  of  possibilities  is  extensive. 

5.  GOPAD  MODEL  DEVELOPMENT  PROCESS 

This  section  describes  how  the  GOPAD  development  software  would  be  used  to 
develop  a  new  weather  forecast  model  for  cloud  cover,  ceiling,  and  visibility  for  an 
overseas  country  where  the  developer  has  no  prior  knowledge  about  the  weather  patterns  and 
where  he  must  create  all  new  historical  data  bases.  As  these  time  estimates  will  show,  the 
majority  of  the  man-hours  required  is  in  formatting,  correcting,  and  organizing  the  historical 
data  bases.  The  development  procedure  would  be  organized  into  three  ordered  phases.  Upon 
completion  of  each  phase,  an  operational  model  would  be  available  that  could  be  fielded. 

5.1  DEVELOP  A  SINGLE-SITE,  SCENARIO  MODEL 

In  Phase  I,  the  objective  would  be  to  develop,  test,  and  deliver  a  single-site,  12-month, 
scenario  model  for  City,  Country  (e.g.,  some  city  in  a  Latin  American  country).  The  model 
developed  in  Phase  I  would  use  either  one,  two,  or  three  sources  of  predictor  variables. 
Obviously,  a  model  that  uses  more  variable  sources  will  take  more  time,  but  its  accuracy 
would  also  be  greater.  So  a  trade-off  decision  must  be  made  by  selecting  from  the  available 
sources  of  predictor  variables  that  may  be  used  to  develop  the  Phase  I  model.  Although  we 
would  fully  expect  the  performance  of  this  simple  model  to  surpass  climatology,  we  cannot 
know  how  much  of  an  increase  is  possible. 

If  only  one  source  of  pmdictor  variables  could  be  used,  then  rawinsonde  data  from 
five  to  ten  stations  in  and  around  the  country  would  be  used.  It  would  take  about  10  weeks  to 
develop  this  model. 

If  two  sources  of  predictor  variables  could  be  used,  then  rawinsonde  and  barotropic 
model  data  would  be  used.  The  historical  barotropic  data  base  would  be  constructed  from 
scratch  for  a  10-year  period.  Obviously,  this  option  will  produce  a  better  forecast  model,  but 
it  will  take  an  additional  effort  to  create  the  barotropic  data  base.  Nevertheless,  the 
production  of  this  data  base  for  a  country  needs  to  be  done  only  once,  but  it  can  be  used  many 
times.  It  would  take  about  26  weeks  to  develop  this  model  from  scratch.  This  option 
probably  offers  the  best  trade-off  between  cost  and  development  time  versus  the  potential  for 
increased  performance. 
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If  three  sources  of  predictor  variables  could  be  used,  then  this  model  would  be 
developed  using  satellite-based  imagery  to  derive  cloud-tracked  winds,  barotropic,  and 
rawinsonde  data  as  predictor  variables.  In  addition  to  the  rawinsonde  data  and  the 
development  of  the  barotropic  historical  data  base,  this  option  would  include  che  development 
of  a  cloud-tracked  winds  historical  data  base  using  GOES  imagery,  CIRRUS  I,  and  a  vortidty 
model  to  derive  synoptic  scale  wind  patterns.  It  would  take  about  45  weeks  to  develop  this 
model  from  scratch. 

5.2  DEVELOP  MULTI-SITE,  MULTI-SOURCE,  SCENARIO  MODEL 

In  Phase  II,  a  multi-site,  multi-source,  scenario  model  would  be  developed  for  up  to 
ten  specified  sites  in  a  country  in  three-hour  increments  out  to  24  hours.  This  model  would 
use  the  predictor  variable  data  sets  produced  in  the  Phase  I  (i.e.,  rawinsonde  only; 
rawinsonde  and  barotropic  only;  or  rawinsonde,  barotropic,  and  synoptic  scale  wind  flow). 

The  multiplexing  concept  would  be  used  to  create  this  model.  Although  the  selection  of  these 
10  sites  may  be  somewhat  arbitrary,  each  site  selected  must  have  about  10  years’  worth  of 
historical  ground  truth  information  available.  It  would  take  about  5  weeks  to  develop  this 
model. 

5.3  DEVELOP  A  HIGH  RESOLUTION,  MULTI-SOURCE,  SCENARIO  MODEL 

In  Phase  ni,  a  high-resolution,  scenario  model  would  be  developed  to  produce 
forecasts  anywhere  in  a  country.  This  model  would  be  constructed  using  all  the  historical 
data  bases  created  in  Phases  I  and  II,  and  at  least  one  new  historical  data  base  of  non- 
meteorological  variables  (e.g.,  terrain  elevation,  direction  and  amount  of  slope,  etc.).  The 
multiplexing  concept  would  be  used  to  create  this  model.  Once  this  model  has  been 
completed,  other  forecast  models  (e.g.,  precipitation,  temperature,  etc.)  would  be  able  to  use 
many  of  Lie  same  historical  data  bases  that  were  developed  for  the  first  time.  Therefore,  the 
costs  of  additional  models  would  be  significantly  less.  It  would  take  about  16  weeks  to 
develop  this  model. 


6.  SUMMARY 

The  Army  and  other  services  need  a  high  technology  initiative  in  mtsoscale  weather 
forecasting  for  high-,  medium-,  and  low-intensity  combat  situations.  Rapid  weather 
forecasting  is  critical  to  the  planning  cycle.  Forecasting,  which  is  a  difficult  task  in  peace¬ 
time,  will  be  even  more  difficult  on  the  battlefield.  The  number  of  weather  parameters 
required  for  a  new  generation  of  Tactical  Decision  Aids  and  automated  systems  has  placed 
increased  importance  on  the  requirement  for  the  Army  to  develop  a  highly  automated, 
tactical  weather  forecasting  system.  New  technologies  and  hardware  are  now  available  that 
may  make  this  possible.  Therefore,  bold  research  is  needed  to  develop  a  technical  foundation 
to  achieve  this  goal. 
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1.0  ABSTRACT 

During  the  summer  of  1989,  the  Forecast  Systems  Laboratory,  NOAA  sponsored  an 
evaluation  of  Artificial  Intelligence  systems  that  forecast  convective  storms  taking  place 
within  a  110-mile  radius  of  Boulder,  Colorado.  This  exercise  was  called  SHOOTOUT-89 
and  was  designed  to  be  an  exploratory  study  of  the  effectiveness  of  a  variety  of  AI 
systems  In  the  weather  forecasting  process.  Data  gathered  during  the  forecast  phase  of 
the  exercise  was  used  to  evaluate  the  performance  of  the  participating  systems.  The 
systems  participating  Included:  (1)  Knowledge  Augmented  Severe  Storms  Predictor 
(KASSPr);  (2)  Goal  Oriented  Pattern  Detection  (GOPAD);  (3)  NOAA/NESDIS  CONVEX;  (4) 
Additive  Linear  Prediction  System  (ALPS);  (5)  WILLARD;  and  (6)  Objective  Convective 
Index  (OCI).  The  goal  of  SHOOTOUT-89  Is  to  provide  AI  system  designers  with  feedback 
so  they  can  design  future  systems  that  will  support  the  overall  weather  forecasting 
process.  This  paper  summarizes  the  results  and  discusses  some  of  the  lessons  learned 
from  this  exercise. 


2.0  INTRODUCTION 

During  the  summer  of  1989,  the  Forecast  Systems  Laboratory  (FSL)  of  the  National 
Oceanic  and  Atmospheric  Administration  (NOAA)  conducted  an  exercise  that  was 
designed  to  evaluate  Artificial  Intelligence  (AI)  systems  that  forecast  significant  and 
severe  weather  over  four  regions  In  the  northeastern  Colorado  plains.  This  exercise, 
called  SHOOTOUT-89,  took  place  in  Boulder,  Colorado.  Previous  real-time  (KI) 
exercises  were  conducted  during  the  summers  of  1983,  1985,  and  1987,  In  northeast 
Colorado  by  the  Program  for  Regional  Observations  and  Forecasting  Services  (PROFS). 
PROFS  operates  an  extensive  network  of  sensors,  maintains  historical  data  bases,  and 
facilitates  advanced  research  studies  In  mesoscale  forecasting. 

The  SHOOTOUT-89  was  an  exploratory,  quantitative,  and  qualitative  comparison  of  AI 
systems  that  forecast  severe  weather.  Although  each  system  required  different 
predictor  variables,  all  systems  received  real-time  data  at  approximately  the  same  time, 
and  were  required  to  produce  a  forecast  for  the  1 1:00  AM.  weather  briefing.  Thus,  the 
unique  format  for  the  SHOOTOUT  exercises  made  it  possible  to  directly  compare  and 
contrast  the  performance  of  «\I*i3uSvd  forecasting  systems  in  a  laboratory’  setting.  Data 
gathered  during  the  exercise  will  be  used  to  evaluate  the  tasks,  rules,  and  regulations  of 
the  study,  as  well  as  to  evaluate  the  participating  systems.  System  evaluation  included 
testing  the  robustness  of  the  systems  In  an  operational  situation,  comparing  the 
effectiveness  of  different  AI  approaches,  determining  the  need  for  expert  human  input, 
evaluating  the  sensitivity  of  a  system  given  different  operators,  soliciting  user 
comments,  and  stimulating  collaboration  among  researchers.  In  this  paper,  we  focus 
on  the  effectiveness  of  the  systems  that  participated. 
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3.0  PARTICIPATING  SYSTEMS 


There  were  six  AI  systems  that  participated  in  the  SHOOTOUT-89  exercise.  A  few  of  the 
systems  required  no  more  than  a  few  keystrokes  to  initiate  the  forecast  model.  Other 
systems  required  extensive  Interaction  with  a  human  expert,  and  some  systems 
interrogated  the  meteorologist.  A  brief  review  of  these  systems  follows: 

1)  KASSPr  (Knowledge  Augmented  Severe  Storm  Predictor)  was  developed  by  the 
Atmospheric  Environment  Service  of  Canada,  Department  of  the  Environment, 
in  cooperation  with  Digital  Equipment  Corporation.  KASSPr  is  a  traditional 
expert  system  written  in  OPS-5.  Rule  firing  is  controlled  by  the  forward¬ 
chaining  OPS-5  production  system.  The  knowledge  base  was  derived  from  John 
Bullas,  an  expert  in  severe  weather  forecasting,  and  Bruno  deLorenzis  (1988), 
the  developer,  from  Atmospheric  Environment  Service  of  Canada. 

KASSPr  requires  both  NGM  numerical  model  output,  and  extensive  input  from 
the  meteorologist  running  the  system.  The  meteorologist  identifies  and  draws 
the  forecasted  positions  of  numerous  meteorological  features,  such  as  troughs 
and  ridges,  on  the  computer  screen  (deLorenzis,  1988).  Once  this  task  has  been 
completed,  the  system  generates  forecasts  without  further  interaction. 

2)  GOPAD  (Goal  Oriented  Pattern  Detection)  was  developed  by  Dr.  Kenneth  Young 
with  the  help  of  Paul  Lampru  of  Consultant's  Choice,  Inc.  GOPAD  is  a 
combination  of  a  front-end  program  which  uses  multiple  discriminant  analysis 
(MDA)  to  screen  a  large  number  of  potential  predictor  variables  and  to  create 
indices,  which  are  linear  combinations  of  these  variables.  The  forecast  model  is 
then  developed  using  a  non-linear  k-nearest  neighbor  approach,  based  on  the 
predictor  variables  selected  by  MDA.  The  forecast  model  was  developed  using 
PROFS  mesonet  data  for  1200Z  through  1555Z,  and  1200Z  rawinsonde  data 
from  Denver  and  seven  surrounding  stations. 

GOPAD  is  designed  to  run  autonomously.  During  the  exercise,  the  necessary 
mesonet  and  rawinsonde  information  was  provided  in  a  file  created  by  the 
PROFS  computer  system,  and  the  forecast  program  was  initiated  by  the  human 
operator.  In  addition  to  the  standard  probability  forecasts  required,  GOPAD 
forecasts  the  probability  of  tornados  and/or  funnel  clouds,  the  probability 
distributions  of  expected  maximum  hailstone  sizes,  and  the  probability 
distributions  of  expected  peak  wind  gusts. 

Two  versions  of  GOPAD  were  run.  The  "learning  version"  received  verification 
data  for  the  previous  day  which,  along  with  the  forecast  parameters  for  that  day, 
were  added  to  its  historical  data  base.  The  "static"  version  received  no  updated 
verification  data  and  so  did  not  alter  the  historical  data  base  from  which  it  made 
forecasts. 

3)  CONVEX  was  developed  by  John  Weaver,  NOAA/NESDIS,  who  provided  the 
severe  storm  forecasting  experience,  and  by  R.  Phillips  from  NOAA/NESDIS,  who 
was  uie  knowledge  engineer  (weaver  and  Phillips,  1987;  Weaver  and  Phillips, 
1989).  CONVEX  relies  heavily  on  a  sounding  analysis  package  which 
determines  the  instability  of  the  host  airmass  and  its  likelihood  of  initiating 
convection  over  the  front  range.  It  also  requires  the  operator  to  predict  the 
surface  temperatures  and  dew  points  for  each  forecast  region,  which  form  the 
basis  for  the  sounding  analysis  package. 


C-2 


In  addition  to  the  Denver  morning  sounding  and  surface  mesonet  data,  CONVEX 
requires  reasonably  knowledgeable  responses  from  a  meteorologist  to  several 
questions  regarding  synoptic  scale  conditions. 

4)  ALPS  (Adaptive  Linear  Prediction  System)  was  developed  by  Tom  Stewart  of  the 
State  University  of  New  York  at  Albany,  and  Cynthia  Lusk  of  the  University  of 
Colorado  at  Boulder.  ALPS  Is  a  linear  model  developed  using  the  methods  of 
judgment  analysis  from  the  field  of  cognitive  science.  The  foundation  for  this 
theory  Is  the  belief  that  linear  models  are  robust  prediction  systems  that  are 
more  consistent  than  human  judgment  in  certain  situations.  This  theory 
suggests  that  simple  algebraic  models  can  capture  the  skill  In  the  judgments  of 
an  expert  In  tasks  that  Involve  a  high  degree  of  uncertainty,  lntercorrelated 
variables,  and  monotonlc  relationships  between  variables  and  the  observed 
event. 

Meteorological  expertise  was  required  only  In  the  selection  of  variables, 
estimating  their  relative  weights,  and  calibrating  the  output.  The  six  key 
variables  used  by  ALPS  are  positive  buoyancy,  wind  shear,  surface  temperature, 
humidity,  wind  direction,  and  wind  speed.  Some  of  the  values  are  read 
automatically  from  the  PROFS  mesonet  data,  while  others  are  determined  by  the 
operator  from  the  Denver  morning  sounding. 

5)  WILLARD  was  developed  originally  by  Steve  Zubrlck  at  Radian  Corporation. 
WILLARD  Is  an  expert  system  where  rules  were  developed  using  an  Induction 
algorithm  In  Rulemaster  (Zubrlck  and  Riese,  1985).  WILLARD  was  originally 
designed  and  developed  to  forecast  the  potential  of  severe  thunderstorms  In  the 
central  United  States.  The  system's  forecasts  are  designed  to  be  similar  to  the 
Convective  Outlooks  issued  three  times  a  day  by  the  National  Severe  Storm 
Forecast  Center. 

WILLARD  is  an  expert  system  that  Is  composed  of  a  hierarchy  of  about  30 
modules  at  least  containing  one  decision  rule.  For  SHOOTOUT-89,  additional 
rules  were  added  to  WILLARD,  based  on  the  expertise  of  forecasters  and  the 
Denver  NWS  Forecast  Office.  WILLARD  might  ask  the  operator  up  to  forty 
questions  before  making  a  forecast.  These  questions  pertain  to  current  synoptic 
and  mesoscale  features,  as  well  as  numerical  forecast  guidance.  Relatively  few 
questions  might  be  asked  If  the  situation  Is  not  iromising  for  severe  weather. 

6)  OCI  (Objective  Convective  Index)  was  developed  by  Robert  Shaw  with 
considerable  Input  from  Thomas  Corona,  Denice  Walker,  and  many  participants 
of  the  PROFS  Real-Time  1987  experiment  (RT-87).  A  long  list  of  potential 
predictor  variables  was  developed,  using  many  Boulder-area  meteorologists  as 
experts.  OCI  uses  proven  severe  weather  forecasting  principles  to  answer  the 
question,  "What  are  the  chances  of  severe  weather  this  afternoon  In  the  PROFS 
mesonet  region?" 

Many  Boulder-area  meteorologists  were  consulted  to  identify  a  comprehensive 
list  of  potential  predictor  variables.  However,  the  amount  of  archived  data  and 
number  of  potential  predictor  variables  were  too  large  for  effective  regression 
equations  to  be  generated.  Consequently,  predictors  were  subjectively  weighted 
to  build  a  linear  model.  Heuristic  rules  were  added  to  identify  relationships 
among  the  variables  that  might  inhibit  convection.  Both  observed  data  and 
NGM  forecast  model  data  were  Included  for  each  of  the  four  basic  weather 
elements:  temperature,  pressure,  moisture,  and  wind.  Since  the  OCI  Is  based 
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on  universal  convective  forecasting  principles,  it  can  be  modified  for  any  climate. 
OCI  icquires  the  operator  to  input  surface  observations,  Denver  sounding  data, 
and  NGM  forecast  model  data  for  Denver  and  Cheyenne.  Forecasts  are 
produced  without  further  interaction. 


4.0  EXERCISE  PERIOD  AND  REGION 

SHOOTOUT-89  ran  from  May  17  through  August  16,  1989.  On  each  day  of  the 
exercise,  all  systems  were  to  generate  forecasts  of  the  mutually  exclusive  and 
exhaustive  probabilities  of  occurrence  of  each  of  three  weather  categories,  In  each  of 
four  designated  zones  In  northeast  Colorado.  The  choice  of  the  forecast  zones  was 
based  on  work  by  Weaver  et  al.,  (1987).  The  four  forecast  regions  are  shown  on  the 
map  In  Figure  1. 


Figure  1.  Forecast  Regions  for  SHOOTOUT-89.  (Roberts  et  al.,  1989) 
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5.0.  DEFINITION  OF  WEATHER  CATEGORIES 


The  weather  categories  are  listed  as  follows: 


Category  0:  Nonsignificant  (Nil)  weather  Is  the  absence  of  category  1  or 
category  2  weather. 

Category  1:  Significant  weather  is  the  absence  of  category  2  weather,  with  the 
occurrence  of  any  of  the  following:  (a)  hall  at  least  0.25  inches  but 
less  than  0.75  Inches  diameter,  (b)  surface  winds  between  35  and 
49  kts,  (c)  a  rainfall  rate  of  2  Inches/hour  or  more,  based  on  5- 
minute  measurements,  and/or  (d)  a  funnel  cloud. 


Category  2:  Severe  weather  is  the  occurrence  of  any  of  the  following:  (a)  hall 
with  a  diameter  of  at  least  0.75  inches,  (b)  wind  gusts  of  50  kts  or 
greater,  and/or  (c)  a  tornado. 


6.0  FORECAST  LEAD  TIME  AND  VALID  PERIOD 

Forecasts  were  required  by  11:15  AM.  (1715Z)  to  be  valid  from  1:00  P.M.  (1900Z)  to 
8:00  P.M.  (0200Z).  Due  to  the  amount  of  manual  input  required,  forecasts  for  OCI  were 
run  In  the  early  afternoon,  based  on  data  available  that  morning.  Not  all  forecast 
models  produced  forecasts  for  all  four  regions  for  all  operational  days  of  the  exercise. 
WILLARD  generated  only  a  single  forecast  for  the  entire  region.  This  forecast  was 
assumed  to  be  the  same  for  each  region.  OCI  did  not  generate  forecasts  for  Region  I. 

Most  of  the  data  required  by  the  systems  was  available  by  10: 15  AM.  Forecasts  for  all 
systems  except  OCI  were  completed  by  11:15  AM.  (1715Z)  and  presented  at  the  FSL 
daily  weather  briefing  at  1 1:25  AM.  Not  all  the  forecast  models  were  operational  on  the 
first  day  of  the  exercise. 


7.0  NUMBER  JF  OPERATIONAL  DAYS 

The  starting  dates  and  number  of  days  each  system  was  operated  are  shown  1’  Table  1. 
Forecasts  were  not  met  when  a  program  crashed  (programming  problems),  or  when 
required  input  data  was  not  available.  OCI  did  not  produce  forecasts  for  Region  I. 
Since  GOPAD  and  ALPS  are  totally  objective  systems,  they  were  rerun  for  the  days  that 
were  missed  during  the  exercise. 
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TABLE  1 

NUMBER  OF  DAYS  EACH  SYSTEM  WAS  OPERATIONAL 

Starting  Date 

#Davs  ODerational 

KASSPr 

17  May  1989 

60  days 

GOPAD  learning 

17  May  1989 

53  days 

GOPAD  static 

24  May  1989 

50  days 

CONVEX 

19  May  1989 

57  days 

ALPS 

7  Jun  1989 

48  days 

OCI 

30  May  1989 

45  days 

WILLARD 

30  May  1989 

54  days 

*  The  number  of  operational  days  for  Regions  II,  III  and  IV. 

After  the  exercise  was  completed,  the  objective  systems  were  allowed  to  generate  offline 
forecasts  for  the  days  they  were  not  operational.  The  purpose  was  to  obtain  the  largest 
number  of  common  days  In  which  forecasts  were  available  for  comparative  evaluations. 
As  a  result  of  this  effort,  there  were  48  common  days  for  Region  I,  and  45  common  days 
for  Regions  n,  HI  and  IV. 


8.0  VERIFICATION  DATA 

8.1  Verification  During  SHOOTOUT-89 

The  collection  of  verification  data  Is  a  crucial  aspect  of  SHOOTOUT-89.  A  full-time 
Verification  Coordinator  (VC)  was  responsible  for  gathering  and  documenting  the 
verification  data  for  the  exercise.  These  data  are  also  used  by  the  Denver  WSFO  In  its 
own  forecast  verification  studies.  The  VC  was  stationed  at  the  Denver  WSFO,  where 
real-time  radar  data  was  readily  available.  When  radar  or  other  data  suggested  possible 
significant  or  severe  weather,  the  VC  called  cooperative  observers  In  the  affected 
regions.  The  VC  also  received  reports  that  were  phoned  In  to  the  Denver  WSFO,  and 
made  follow-up  phone  calls  on  the  day  following  a  possible  weather  event. 

The  following  sources  provided  verification  data:  (1)  a  volunteer  spotter  network,  and  a 
paid  cooperative  observer  network  sponsored  by  the  NWS;  (2)  police  and  fire  stations, 
county  emergency  preparedness  staffs,  and  highway  road  crews;  (3)  a  network  of 
amateur  radio  operators;  (4)  weather  service  offices  In  Colorado  Springs  and  Cheyenne, 
In  addition  to  the  Denver  WSFO;  (5)  automated  mesonet  stations  (PROFS)  that  provide 
information  on  wind  gusts  and  rainfall  rates;  (6)  dally  weather  observations  recorded  by 
a  network  of  approximately  30  specially  recruited  weather  observers  with  observations 
mailed  in  monthly;  and  (7)  volunteer  chase  teams  consisting  of  research  meteorologists 
who  maintain  contact  with  the  VC  by  cellular  phones,  or  who  report  after  the  fact.  In 
spite  of  this  extensive  list  of  possible  observers,  It  was  nonetheless  very  difficult  to 
obtain  verification  data  with  sufficient  accuracy  to  make  absolute  measures  of  skill  or 
performance  highly  reliable. 
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8.2  Statlonarltv  of  Historical  Validation  Data 


Table  2  contrasts  the  frequencies  of  severe  and  significant/severe  weather  events 
observed  In  1989  with  those  observed  In  previous  years  for  each  of  the  four  forecast 
regions.  With  the  exception  of  Region  I,  the  observed  frequencies  of  severe  weather  for 
1989  were  similar  to  the  frequencies  observed  for  three  previous  exercises.  Table  2  also 
suggests  that  significant  weather  was  greatly  over-reported  in  1989,  or  at  least 
significantly  different  than  during  1985  and  1987.  Note  that  we  did  not  have  access  to 
significant  weather  Information  collected  In  1983.  The  observed  frequencies  In  1989  are 
probably  more  reflective  of  the  true  climatology  for  each  region.  The  probability  that  the 
observed  frequencies  for  1989  occurred  by  chance,  given  the  averages  for  1985  and 
1987,  Is  less  than  0.1%;  that  is,  this  Is  not  a  chance  occurrence.  Obviously,  the 
methods  of  reporting  significant  weather  events  dramatically  changed  In  1989  as 
compared  to  previous  years. 


TABLE  2 

OBSERVED  FREQUENCY  OF  EVENTS  BY  REGION 
Severe  Weather  Frequency 


I 

n 

Iff 

rv 

Observed 

(1989) 

8.3% 

4.4% 

13.3% 

17.8% 

Observed 

(1983,1985.1987) 

2.9% 

5.5% 

8.8% 

18.4% 

Significant  or  Severe  Weather  Frequency 

I 

n 

in 

IV 

Observed 

(1989) 

45.8% 

37.8% 

33.3% 

55.6% 

Observed 

(1985,1987) 

9.4% 

14.9% 

15.8% 

29.2% 

Serious  over-reporting,  (or  under-reporting),  of  significant  weather  adversely  affects  the 
performance  of  all  the  models,  but  it  especially  impacts  GOPAD  and  ALPS,  since  the 
historical  data  sets  for  RT-83,  RT-85,  and  RT-87  were  essential  to  the  development  of 
these  models.  Although  this  situation  might  occur  when  tactical  forecast  models  are 
developed  for  one  region  and  used  In  another  region,  or  whenever  one  year  Is  abnormal 
from  all  other  years,  it  should  not  vary  so  much  that  it  Is  statistically  significant.  Thus, 
the  possibility  that  past  exercises  seriously  under-reported  significant  weather  Is  a 
critical  Issue  in  properly  comparing  the  performance  of  all  the  models  that  participated 
in  the  SHOOTOUT-89  exercise.  In  order  to  investigate  this  issue  In  depth,  a  Markov 
chain  simulation  was  performed  to  compare  the  reporting  of  significant  and  severe 
weather  events  In  1985  and  1987  to  1989. 

8.3  Markov  Chain  Simulation 

The  fraction  of  significant  weather  events  reported  during  the  RT-89  experiment,  shown 
In  Table  3,  Is  markedly  higher  tha  the  fraction  reported  during  the  RT-85  and  RT-87 
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experiments  as  shown  In  Table  4.  The  fraction  of  Nil  weather  events  was  likewise  much 
lower  in  1989.  However,  the  fraction  of  severe  weather  events  appears  to  be  quite 
similar. 


TABLE  3 

1989  REPORTED  NIL,  SIGNIFICANT.  AND  SEVERE  WEATHER 

Nil 

Sig 

Sev 

I 

35  (56%) 

23  (37%) 

4  (6%) 

62 

n 

39  (63%) 

20  (32%) 

3  (5%) 

62 

m 

42  (68%) 

13  (21%) 

7  (11%) 

62 

IV 

31  (50%) 

20  (32%) 

11  (18%) 

62 

Sum 

147  (59%) 

76  (31%) 

25  (10%) 

248 

TABLE  4 

1985  AND  1987  REPORTED  NIL,  SIGNIFICANT,  AND  SEVERE  WEATHER 


Nil 

Sig 

Sev 

I 

183  (91%) 

15(7%) 

4  (2%) 

202 

n 

172  (85%) 

18(9%) 

12  (6%) 

202 

m 

170  (84%) 

14(7%) 

18  (9%) 

202 

IV 

143  (71%) 

23(11%) 

36  (18%) 

202 

Sinn 

668  (83%) 

70(9%) 

70  (9%) 

808 

Note:  Frequencies  may  vary  from  those  In  Table  2  because  only  1985  and  1987 

severe  events  are  Included  In  this  Table. 


This  situation  raises  a  question:  Is  the  larger  fraction  of  significant  weather  reports 
during  1989  within  expected  statistical  fluctuations?  In  order  to  answer  this  question, 
Markov  chain  simulations  were  conducted  for  each  of  the  four  regions.  These 
simulations  assumed  that  the  probabilities  of  significant  or  severe  weather  are 
influenced  by  the  occurrence  or  non-occurrence  of  significant  or  severe  weather  on  the 
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The  Null  hypothesis  assumed  the  true  population  of  significant  and  severe  weather 
events  was  that  observed  in  the  RT-85  and  RT-87  experiments.  The  transition  matrices 
used  for  the  Markov  chain  simulations  are  shown  in  Table  5.  In  region  IV,  for  example, 
the  transition  probability  for  a  severe  weather  occurrence  to  occur  on  the  next  day. 
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given  a  significant  weather  event,  Is  0.250.  The  transition  probabilities  were  assumed 
to  be  stationary;  that  Is,  they  do  not  change  throughout  the  experiments  (from  May 
through  August). 


TABLE  5 

MARKOV  CHAIN  TRANSITION  PROBABILITIES 


Region  I 

Region  n 

Nil 

Sig 

Sev 

Nil 

Sig 

Sev 

Nil 

.922 

.061 

.017 

Nil 

.860 

.087 

.052 

Sig 

.733 

.200 

.067 

Sig 

.765 

.176 

.059 

Sev 

1.000 

.000 

.000 

Sev 

.833 

.000 

.167 

Region  III 

Region  IV 

Nil 

Sig 

Sev 

Nil 

Sig 

Sev 

Nil 

.851 

.065 

.083 

Nil 

.710 

.099 

.191 

sig 

.786 

.143 

.071 

Sig 

.583 

.167 

.250 

Sev 

.778 

.056 

.167 

Sev 

.722 

.139 

.139 

The  RT-89  experiment  was  conducted  on  62  days  within  a  92-day  span  from  May  17 
through  August  16.  Thus,  a  simulated  sequence  of  92  days  was  conducted  and  then 
analyzed  by  removing  the  30  days  on  which  the  RT-89  experiment  was  not  conducted. 
Each  sequence  was  Initiated  using  the  stationary  probabilities  for  Nil,  Significant,  and 
Severe  weather.  The  number  of  Nil,  Significant,  and  Severe  weather  days  was 
determined  for  each  of  10,000  trials.  Cumulative  distributions  of  the  fraction  of  trials 
that  yielded  "n"  or  fewer  Nil,  Significant,  and  Severe  days  were  prepared. 

These  cumulative  distributions  were  developed  for  each  of  the  four  regions.  The 
probabilities  that  the  fractions  of  Nil,  Significant,  and  Severe  weather  events  reported 
for  1989  could  have  occurred  by  chance  (P-values),  given  the  transition  probabilities 
based  on  the  1985  and  1987  experiments,  are  readily  derived  from  these  cumulative 
distributions. 

For  example.  Table  6  shows  the  cumulative  distribution  of  the  fraction  of  trials  that 
yielded  "n"  or  fewer  Nil,  Significant,  and  Severe  weather  events  for  Region  IV  based  on 
the  Markov  chain  simulations.  During  1989,  there  were  31  Nil  weather  events  reported 
lor  Region  IV.  Using  the  cumulative  distribution  shown  In  Table  6,  the  probability  ol 
having  31  or  fewer  Nil  weather  events  Is  0.12%.  Similarly,  the  probability  of  having  20 
or  more  significant  weather  events  Is  less  than  0.01%,  and  the  probability  of  having  11 
or  fewer  severe  weather  events  is  49%.  Thus,  we  conclude  that  there  Is  no  difference  In 
the  frequency  of  severe  weather  events  reported  for  Region  IV,  but  the  frequency  of 
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significant  weather  events  Is  significantly  higher  and  the  frequency  of  Nil  weather  events 
Is  significantly  lower  in  1989. 


TABLE  6 

CUMULATIVE  DISTRIBUTIONS  FROM  MARKOV  CHAIN  SIMULATIONS 

FOR  REGION  IV 


n _ Nil _ Slg _ Sev _ n _ Nil 


0 

0.00% 

0.02% 

0.00% 

31 

0.12% 

1 

0.00 

0.51 

0.00 

32 

0.23 

2 

0.00 

2.57 

0.02 

33 

0.50 

3 

0.00 

7.24 

0.05 

34 

1.17 

4 

0.00 

15.68 

0.37 

35 

2.05 

5 

0.00 

27.55 

1.23 

36 

3.84 

6 

0.00 

42.73 

3.31 

37 

6.50 

7 

0.00 

57.67 

7.39 

38 

10.07 

8 

0.00 

71.63 

14.18 

39 

15.37 

9 

0.00 

82.24 

23.39 

40 

22.57 

10 

0.00 

89.83 

36.09 

41 

31.66 

11 

0.00 

94.58 

49.06 

42 

42.19 

12 

0.00 

97.53 

62.32 

43 

52.65 

13 

0.00 

98.86 

74.23 

44 

62.81 

14 

0.00 

99.59 

83.37 

45 

72.37 

15  . 

0.00 

99.84 

89.90 

46 

81.07 

16 

0.00 

99.94 

94.22 

47 

87.75 

17 

0.00 

99.97 

97.20 

48 

92.62 

18 

0.00 

99.99 

98.67 

49 

95.90 

19 

0.00 

99.99 

99.40 

50 

‘  ''8 

20 

0.00 

100.00 

99.80 

51 

l.  04 

21 

0.00 

99.94 

52 

99.65 

22 

0.00 

99.99 

53 

99.84 

23 

0.00 

99.99 

54 

99.94 

24 

0.00 

100.00 

55 

99.99 

25 

0.00 

56 

99.99 

26 

0.00 

57 

100.00 

27 

0.00 

58 

100.00 

28 

0.01 

59 

100.00 

29 

0.01 

60 

100.00 

30 

0.03 

61 

100.00 

The  P-values  representing  the  probabilities  that  the  numbers  of  Nil,  Significant,  and 
Severe  weather  events  reported  in  1989  are  within  the  expected  range  of  natural 
variability  based  on  the  1985  and  1987  experiments,  as  shown  In  Table  7.  The  pattern 
discussed  above  for  Region  IV  Is  quite  similar  for  the  other  three  regions.  Region  I  does 
show  an  increased  frequency  of  severe  weather  events  in  addition  to  the  Increased 
frequency  of  significant  weather  events. 
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TABLE  7 

P-VALUES  LIKELIHOOD  RANGE 


Nil 

Sig 

Sev 

I 

<0.01% 

<0.01% 

3.3% 

II 

<0.01% 

<0.01% 

50% 

m 

0.20% 

0.03% 

33% 

IV 

0.12% 

<0.01% 

50% 

8.4  Conclusions 

Therefore,  it  may  be  concluded  that  the  observed  frequency  of  severe  weather  events  for 
1989  Is  within  the  range  of  natural  variability  given  by  the  1985  and  1987  observations, 
whereas  the  frequency  of  significant  weather  events  for  isf89  is  certainty  not  within  the 
expected  range.  This  situation  suggests  that  the  verification  procedures  used  in  RT-89 
were  significantly  different  than  those  used  in  RT-85  and  RT-87  in  detecting  the 
occurrence  of  significant  weather.  Note  that  the  Increase  in  significant  weather  events 
is  balanced  by  a  decrease  in  the  Nil  weather  events  with  virtually  no  effect  on  the  severe 
weather  events.  This  would  not  be  expected  if  the  changes  were  due  to  natural 
variability. 

The  primary  difference  between  the  RT-85/RT-87  exercises  and  the  RT-89  exercise 
appears  to  be  the  method  of  validating  severe  and  significant  weather  events.  For 
example,  the  RT-85/RT-87  exercises  used  chase  teams  to  verify  severe  weather  events 
while  none  were  used  during  RT-89.  These  chase  teams  confined  their  efforts  to 
Regions  n,  HI,  and  IV  and  did  not  pursue  storms  in  Region  I.  During  RT-89,  the 
validation  coordinator  actively  used  radar  information  to  identify  where  severe  and 
significant  weather  might  be  occurring.  Aggressive  telephone  calls  were  then  initiated 
to  sheriffs  departments,  fire  departments,  highway  crews,  and  other  agencies  located  in 
areas  where  the  radar  showed  activity. 

These  differences  suggest  that  the  RT-89  verification  methods  were: 

(a)  equivalent  to  using  chase  teams  to  determine  the  incidence  of  severe 
weather  in  Regions  II,  III,  and  IV; 

(b)  superior  to  the  chase  teams  in  determining  the  incidence  of  severe  weather 
in  Region  I; 

(c)  superior  to  the  chase  teams  in  determining  the  incidence  of  significant 
weather  in  all  regions. 


9.0.  PERFORMANCE  COMPARISONS 

All  forecasts  models  that  participated  in  the  SHOOTOUT-89  exercise  produced  three 
forecasts-the  probability  of  Nil  weather,  significant  weather,  and  severe  weather-for 
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each  day  for  each  region,  except  OCI  which  did  not  forecast  for  Region  I.  Evaluations 
were  done  for  Nil  versus  Non-Nil.  and  severe  versus  non-severe  weather. 

The  GOPAD  forecast  models  used  In  RT-89  were  developed  on  the  basis  of  reports  of 
significant  and  severe  weather  reports  obtained  in  RT-83,  RT-85,  and  RT-87.  There 
were  virtually  no  reports  of  significant  weather  available  for  RT-83.  The  Incidence  of 
significant  weather  reports  for  RT-85  and  RT-87  were  substantially  lower  than  in  RT-89. 
Thus,  one  may  expect  that  the  skill  scores  for  GOPAD  based  on  the  significant  weather 
forecasts  will  be  much  lower  than  If  the  "training"  data  had  shown  a  similar  frequency 
of  significant  weather  events.  Likewise,  the  skill  scores  for  severe  weather  In  Region  I 
should  be  adversely  affected  as  well. 

9.1  Skill  Scores 

The  individual  forecasts  for  each  system,  for  each  region,  and  for  each  of  the  common 
days  for  which  all  systems  produced  forecasts,  are  presented  In  Attachments  1  to  9. 
The  event  outcome  for  each  day  is  coded  as  follows:  EO  means  Nil  weather;  El  means 
significant  weather;  and  E2  means  severe  weather.  The  Brier  scores  and  the  Brier- 
based  skill  scores  are  also  given  for  each  region  and  each  system. 

9.1.1  Regional  Climatological  Brier  Scores 

The  Brier  score  Is  a  cumulative  form  where  the  probability  that  the  event  will  occur  Is  P. 
If  the  event  occurs,  then  the  contribution  to  the  Brier  score  is  1  -  (1-P)2.  If  the  event 
does  not  occur,  then  the  contribution  to  the  Brier  score  Is  1-P2.  The  climatological 
Brier  score  is  determined  In  the  same  manner  using  the  long-term  frequency  as  the 
forecast  probability  {1983,  1985,  and  1987  for  severe  weather;  1985  and  1987  for 
significant  weather).  The  long-term  frequencies  on  which  the  climatological  Brier  scores 
are  based  are  listed  In  Table  8. 


TABLE  8 

LONG-TERM  FREQUENCIES  USED  FOR 
CLIMATOLOGICAL  BRIER  SCORE  CALCULATIONS 

I  11 

III 

IV 

Non-Nil 

Severe 

9.41%  14.85% 

3.68%  5.52% 

15.84% 

8.82% 

29-21% 

iS.38% 

9.1.2  Brier-Based  Skill  Scores 

A  Brler-baser4  'kill  score  is  calculated  from  the  observed  Brier  score  for  the  actual 
forecasts  and  *  s  climatological  Brier  score.  Attachments  1  to  4  show  the  calculation  of 
Brier  scores  for  severe  weather.  Attachments  1  to  8  show  the  Brier  scores  for  severe 
and  non-Nil  weather.  For  example.  Attachment  4,  severe  weather  forecasts  for  Region 
IV,  shows  the  Brier  score  for  GOPAD  1  to  be  39.332.  Using  the  long-term  frequencies 
for  those  same  forecasts,  the  climatological  Brier  score  Is  38.421.  The  maximum 


C-12 


possible  Brier  score  Is  45  (i.e.,  no  error  In  any  of  the  forecasts).  Thus,  the  Brier- based 
skill  score  is  calculated  as  follows: 

SS  =  (39.332  -  38.421)  /  (45  -  38.421)  =  +13.9%. 

This  formulation  of  the  Brier  score  makes  it  easy  to  combine  the  scores  from  each  of  the 
four  regions  to  determine  the  overall  forecast  skill.  Attachment  9  shows  the  Brier 
scores  for  (1)  severe  weather,  (2)  non-Nil,  and  (3)  combined  forecasts.  For  example,  the 
Brier  scores  for  GOPAD1  in  each  of  the  four  regions  are  44.019,  43.012,  39.164,  and 
39.332,  totaling  165.527.  The  corresponding  climatological  Brier  scores  are  44.229 
43.084,  39.709,  and  38.421,  totaling  165.443.  The  corresponding  perfect  score  is  48  + 
45  +  45  +  45,  totaling  183.  Thus,  the  severe  weather  skill  score  for  the  four  regions 
combined  is  calculated  as  follows: 

SS  =  (165.527  -  165.443)  /  (183-165.443)  =  +  0.5% 

The  analysis  of  severe  weather  forecasts  appears  to  be  a  better  indicator  of  the 
forecasting  skill  than  an  analysis  that  is  based  upon  non-Nil  weather  forecasts.  This 
opinion  is  based  upon  the  fact  that  there  appears  to  be  a  significant  difference  in  the 
observed  frequency  of  significant  weather  in  1985  and  1987  compared  to  1989.  It  is 
difficult  to  reconcile  this  difference  when  no  similar  difference  appears  in  severe 
weather. 

9.1.3  Conclusions 

There  are  many  different  ways  to  compare  the  performance  of  the  system  that 
participated  in  the  SHOOTOUT-89  exercise.  We  choose  to  rank  the  models  in  two  ways. 

First,  Table  9  shows  the  (Brier)  skill  scores  for  severe  weather  based  upon  the  forecasts 
in  which  there  was  no  statistically  significant  difference  between  reporting  of  severe 
weather  between  1985/87  and  1989  (i.e..  Regions  n,  in,  and  IV).  This  table  eliminates 
the  effect  from  seriously  over-reporting  significant  weather  In  all  regions  and  from  over¬ 
reporting  severe  weather  in  Region  I.  GOPAD  and  ALPS  should  be  helped  by  this  caveat 
since  they  were  trained  on  the  1985  and  1987  data. 


TABLE  9 

SKILL  SCORES  FOR  SEVERE  WEATHER 
IN  REGIONS  ff,  m,  AND  IV 


GOPAD  1 

+  2.1% 

GOPAD2 

+  1.9% 

ALPS 

-  0.7% 

WILLARD 

-11.9% 

KASSPr 

-17.0% 

CONVEX 

-24.0% 

OCI 

-32.3% 

Second,  Table  10  shows  the  skill  scores  only  for  Region  IV,  which  was  the  only  region 
that  had  an  adequate  number  of  events  upon  which  to  train  a  GOPAD  model.  This 
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table  mitigates  the  lack  of  a  large  historical  data  base  that  is  needed  In  order  to  develop 
a  GOPAD  or  ALPS  model. 


TABLE  10 

SKILL  SCORES  FOR  SEVERE  WEATHER 
IN  REGION  IV 


GOPAD  1 

+13.8% 

GOPAD2 

+  13.2% 

ALPS 

+  1.8% 

KASSPr 

-  0.2% 

OCI 

-3.1% 

WILLARD 

-12.6% 

CONVEX 

-40.9% 

9.2  Relative  Operating  Characteristic 

A  good  forecast  system  should  be  able  to  ’detect"  or  correctly  forecast  the  event  without 
giving  too  many  "false  alarms."  Consider  the  following  tables  for  GOPAD  1  forecasts  of 
severe  weather  for  Region  IV.  Consider  that  forecast  probabilities  above  the  threshold 
(18%  and  40%,  respectively}  represent  a  forecast  for  severe  weather,  whereas  forecast 
probabilities  below  the  threshold  are  forecasts  that  severe  weather  will  not  occur. 


7'ABLE  Ii 

SEVERE  WEATHER  FORECASTS  FOR  REGION  IV 
(GOPAD  1) 


Pr  Pr  Pr  Pi’ 

<=18%  >18%  Sum  <=40%  >40%  Sum 


EO/E1 

27 

10 

37 

EO/EI 

35 

2 

37 

E2 

3 

5 

8 

E2 

6 

2 

8 

Sum 

30 

15 

45 

Sum 

41 

4 

45 

Consider  the  first  table  with  a  threshold  of  18%.  GOPAD  1  forecast  severe  weather  on 
15  occasions  with  severe  weather  occurring  5  times.  Note  chat  five  of  the  eight  severe 
weather  occurrences  were  "detected,"  or  the  probability  of  dete*.  Ung  severe  weather 
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(POD)  Is  5/8  or  0.625.  There  were  ten  forecasts  for  severe  weather  which  were  not 
followed  by  severe  weather,  l.e.,  false  alarms.  The  false  alarm  rate  (FAR)  Is  10/37,  or 
0.270. 

One  may  improve  (decrease)  the  FAR  by  increasing  the  probability  threshold.  If  this 
threshold  is  increased  to  40%  (Table  10),  the  FAR  drops  to  2/37,  or  0.054.  The 
probability  of  detecting  severe  weather  events  also  decreases  (i.e.,  the  forecast  will  miss 
more  occurrences  of  severe  weather).  Now  the  POD  is  2/8,  or  0.250. 

The  relative  operating  characteristic  (ROC)  combines  t^e  FAR  and  POD  into  a  single 
diagram  over  the  entire  range  of  thresholds.  A  forecast  system  with  positive  skill  will 
consistently  show  a  larger  POD  than  a  FAR.  The  ROC  plot  will  lie  above  the  diagonal 
between  (0,0)  and  (1,1).  The  further  above  the  diagonal  the  ROC  plot  lies,  the  better  the 
discrimination  between  severe  and  non-severe  weather.  Note  that  this  measures  the 
discrimination  ability  of  the  forecast  model  and  it  is  not  affected  by  bias. 

The  ROC  plots  shown  in  Figures  4  and  5  show  a  much  smaller  spread  among  the 
different  forecast  systems  than  do  the  ROC  plots  in  Figures  2  and  3.  Clearly,  this 
difference  is  a  result  of  using  a  larger  number  of  forecasts  to  determine  the  ROC  plot. 
The  significant  +  severe  weather  (Figure  4)  ROC  plots  suggest  CONVEX,  KASSPr,  and 
GOPAD  exhibit  the  greatest  skill,  whereas  WILLARD  and  ALPS  demonstrate  the  least 
skill,  with  substantial  portions  of  their  ROC  plots  lying  below  the  diagonal  These 
findings  are  not  inconsistent  with  the  conclusions  given  by  the  Brier  score  analysis. 

The  forecast  systems  did  better  forecasting  severe  weather,  according  to  Figure  5.  Here, 
only  WILLARD  shows  a  portion  of  its  ROC  plot  below  the  diagonal.  GOPAD  exhibits 
relatively  smooth  ROC  plots,  suggesting  it  produces  useful  forecasts  over  its  entire 
range.  The  OCI  forecast  system  does  a  much  better  job  in  the  upper  right  region  of  its 
ROC  plot,  suggesting  it  does  a  better  job  identifying  those  days  on  which  severe  weather 
will  occur  (lower  left)  somewhat  better  than  it  identifies  days  on  which  severe  weather 
will  not  occur. 

9.3  Measure  of  Forecast  Sharpness 

A  sharp  forecast  model  tends  to  issue,  for  example,  either  rain  or  no-rain  forecasts. 
Obviously,  the  ideal  forecast  model  would  be  very  sharp  (ie.,  totally  categorical)  and 
perfectly  accurate.  However,  such  a  perfect  model  is  far  beyond  current  technology. 

Since  the  state  of  the  art  in  weather  forecasting  is  far  less  than  perfect,  a  non- 
categorical  or  probabilistic  model  is  the  best  means  for  providing  information  on- 
demand  to  a  myriad  of  Tactical  Decision  Aids  fTDA).  If  a  TDA  is  designed  to  use  a 
categorical  type  of  forecast,  then  that  TDA  should  receive  the  probabilistic  forecast  and 
convert  it  to  a  categorical  forecast,  based  upon  a  criteria  tailored  for  its  individual 
tolerance  for  false  alarms,  and/or  probability  of  detecting  the  event.  Each  TDA  may  be 
expected  to  have  its  own  tolerance  level,  and  thu3  its  own  interpretation  of  the 
probabilistic  forecast. 

The  distribution  of  forecast  probabilities  for  each  of  the  models  is  presented  in  Tables 
12,  13,  and  14.  The  models  are  arranged,  in  Increasing  order  of  sharpness,  from  ALPS, 
which  produces  forecasts  that  deviate  little  from  climatology,  to  KASSPr,  which 
produces  nearly  categorical  forecasts.  Forecasts  that  are  overly  sharp  may  be  expected 
to  show  decreases  in  skill. 
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SEVERE 
REGION  IV 


KASSPr 

GOPAD1 

CONVEX 

ALPS 

OCI 

WILLARD 


*  -0.2% 
-+13.8% 
.-409% 
-  +1.8% 
-  -3.1% 
-126% 


SIG  &  SEV 
REGION  I 


KASSPr 

GOPAD1 

CONVEX 

ALPS 

WILLARD 


—  +5.2% 

—  +8.0% 
— +14.4% 

- 18.0% 

—  -7.0% 


Figure  2.  Relative  Operating  Characteristic  Figure  3.  Relative  Operating  Characteristic  for 

for  Severe  Weather  in  Region  IV  Significant  and  8evere  Weather  in  Region  I 


Figure  4.  Relative  Operating  Characteristic  for  Figure  5.  Relative  Operating  Characteristic 
Significant  and  Severe  Weather  in  for  Severe  Weather  in  All  Regions 

All  Regions 

•  n  i 


GOPAD,  ALPS,  and  OCI  were  capable  of  producing  virtually  any  probability  between  0 
and  100%;  KASSPr,  CONVEX,  and  WILLARD  were  rule-based  systems  that  provided  for 
a  relatively  limited  number  of  options  for  probability  forecasts.  To  a  certain  extent,  the 
three  models  with  the  greatest  sharpness,  CONVEX,  KASSPr,  and  WILLARD,  were 
designed  to  be  categorical  in  nature  because  they  were  intended  to  assist  human 
forecasters  in  issuing  severe  weather  watches.  Tables  12  and  13  list  the  systems  in 
increasing  order  of  sharpness. 


TABLE  12 

DISTRIBUTION  OF  SEVERE  FORECASTS 
ALL  REGIONS  COMBINED 

Forecasted  Probability  of  Severe  Weather 

Sharp- 


Model 

0-9% 

10-19% 

20-39% 

40-69% 

70-100% 

Total 

ness 

ALPS 

76% 

24% 

192 

0.22 

GOPAD 

73% 

15% 

8% 

3% 

212 

0.30 

OCI 

56% 

11% 

18% 

11% 

4% 

129 

0.43 

WILLARD 

80% 

15% 

6% 

216 

0.80 

CONVEX 

80% 

13% 

2% 

5% 

228 

0.83 

KASSPr 

93% 

1% 

1% 

1% 

4% 

240 

0.95 

TABLE  13 

DISTRIBUTION  OF  NON-NIL  FORECASTS 
ALL  REGIONS  COMBINED 

Forecasted  Probability  of  Non-Nil  Weather 

Sharp- 


Model 

0-9% 

10-19% 

20-39% 

40-69% 

70-100% 

Total 

ness 

ALPS 

54% 

34% 

11% 

192 

0.19 

GOPAD* 

41% 

33% 

18% 

7% 

1% 

212 

0.22 

OCI 

21% 

27% 

21% 

21% 

10% 

129 

0.31 

WILLARD 

50% 

29% 

17% 

Wo 

216 

0.56 

CONVEX 

40% 

19% 

20% 

3% 

18% 

228 

0.64 

KASSPr 

84% 

1% 

5% 

10% 

240 

0.93 
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Another  desirable  characteristic  for  a  weather  forecasting  model  is  that  the  model 
should  produce  forecasts  whose  average  for  a  season  Is  veiy  close  to  the  observed 
frequency  of  the  event  being  forecasted  (i.e.,  an  unbiased  model).  Bias  Is  defined  as  the 
average  event  forecast  probability  divided  by  the  observed  event  frequency.  A  bias  less 
than  one  indicates  under-forecasting  (l.e.,  forecasted  probabilities  are  too  low);  whereas 
a  forecast  greater  than  one  Indicates  over-forecasting. 

Tables  14  and  15  show  the  forecast  bias  for  each  region  and  the  overall  bias  for  each 
forecast  model.  The  models  in  Tables  14  and  15  are  listed  In  order  from  least  overall 
bias  to  greatest  overall  bias.  Since  GOPAD  was  trained  and  optimized  on  the  observed 
frequencies  for  the  past  three  exercises,  GOPAD  should  have  very  little  bias,  unless 
ground  truth  reporting  methods  are  altered.  OCI  was  the  only  system  that  conslstantly 
over-predicted  severe  weather.  All  the  other  systems  tended  to  under-predict  both  non- 
Nil  and  severe  weather. 

Bias  in  a  model's  forecast  may  be  expected  to  decrease  the  skill  scores.  If  the  forecast  is 
determined,  the  model's  forecasting  skill  can  be  Improved  by  correcting  for  the  bias. 
The  model  with  the  greatest  bias  tend  also  to  be  the  models  exhibiting  the  least  skill 
(Le.,  KASSPr  and  WILLARD). 


TABLE  14 

SEVERE  WEATHER  FORECAST  BIASES 

Overall 


I 

n 

m 

IV 

Bias 

GOPAD2 

.52 

1.04 

.64 

1.02 

0.812 

GOPAD  1 

.48 

1.04 

.63 

1.01 

0.798 

OCI 

X 

3.06 

1.40 

1.20 

(1.509) 

CONVEX 

.40 

.70 

.75 

.43 

0.569 

ALPS 

.04 

.88 

.47 

.75 

0.544 

KASSPr 

.86 

1.06 

.06 

.12 

0.339 

WILLARD 

.40 

.65 

.22 

.16 

0.273 

•OCI  Is  ranked  by  1/1.509  since  there  Is  an  Inverse  relation  between  under-forecasting  and 
over-forecasting. 
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TABLE  15 

NON-NIL  FORECAST  BIASES 

Overall 


I 

n 

in 

IV 

Bias 

CONVEX 

.45 

.78 

1.09 

.53 

0.668 

OCI 

X 

.59 

.95 

.66 

0.714* 

GOPAD2 

.28 

.29 

.47 

.47 

0.382 

GOPAD1 

.27 

.29 

.47 

.46 

0.372 

JVILLARD 

.30 

.34 

.39 

.23 

0.304 

KASSPr 

.38 

.34 

.29 

.17 

0.284 

ALPS 

.02 

.22 

.31 

.36 

0.229 

•GCI  Is  ranked  by  comparison  to  biases  for  the  other  systems  based  on  Regions  n,  HI,  and  IV 
(non-show). 


10.0  CONCLUSIONS 

Accurate  verification  of  severe  and  significant  weather  events  plays  a  very  Important 
role  In  evaluating  the  performance  of  those  forecast  models.  Verification  of  weather 
events  for  a  region  rather  than  a  single  point  depends  upon  a  network  of  diligent  and 
cooperative  observers.  The  verification  data  received  for  RT-89  reflects  a  substantial 
change  In  verification,  especially  In  recognizing  significant  weather.  This  change  may 
be  a  result  of  a  more  active  procedure  for  gathering  verification  data. 

The  valid  periods  and  ref^ns  for  the  SHOOTOUT-89  exercise  did  not  match  those  used 
for  operational  fore^^ers  by  the  Denver  WSFO,  so  direct  comparisons  of  performance 
to  the  forecasts  h..  ed  by  operational  meteorologists  were  not  possible.  One 
recommendation  for  future  Improvement  In  the  format  for  the  SHOOTOUT  exercises  Is 
that  a  human  forecaster  should  be  used  as  a  benchmark  so  that  it  might  be  possible  to 
estimate  the  potential  these  Al-based  systems  offer  for  Improving  human  forecasting 
skill. 

Another  desirable  characteristic  is  that  a  tactical  forecast  model  should  forecast  each 
event  individually  (e.g.,  hail  size,  high  winds,  funnel  cloud/tomado)  rather  than 
forecasting  a  weather  phenomenon  (e.g.,  convection-induced  severe  or  significant 
weather)  that  Is  a  combination  of  events.  Clearly,  the  Attack  Helicopter  Battalion  is 
interested  in  forecasts  of  high  winds,  whether  they  are  convection  induced  or  non¬ 
convection  Induced. 

In  addition,  probability  forecasts  should  also  include  a  confidence  level  to  measure  the 
degree  of  certainty  In  the  forecast  and,  thereby,  enable  a  TDA  to  provide  even  more 
tailored  information.  Although  none  of  the  models  had  this  feature,  it  is  an  Important 
feature  that  should  be  required  in  tactical  forecast  models. 

The  rich  variety  of  forecast  models  has  provided  a  source  of  information  that  makes  it 
possible  to  better  understand  the  nature  of  our  forecast  modeling  approach.  We  have 
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provided  the  reader  with  a  general  overview  of  how  the  SHOOTOUT-89  exercise  was 
conducted. 

The  veiy  fact  that  six  Al-systems  were  brought  together  in  one  location  to  make  the 
same  forecasts  on  the  same  day  in  an  operational  setting,  and  that  each  system 
produced  forecasts  for  a  majority  of  the  operational  days,  is  sufficient  to  deem  the  RT- 
89  experiment  a  success.  The  nature  of  SHOOTOUT-89  made  it  possible  for  different 
developers  to  discuss  their  approaches  in  a  veiy  cooperative  spirit.  The  SHOOTOUT 
exercises  are  important  to  advancing  the  state  of  the  art  in  mesoscale  event  forecasting. 
The  exceptional  effort  provided  by  NOAA/FSL,  the  chief  meteorologist.  Woody  Roberts, 
and  Bill  Moninger  are  major  factors  in  this  success. 
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ABSTRACT 

A  LISP  program  called  Cloud  Image  Representation,  Recognition,  and 
Understanding  Software  (CIRRUS  I)  autonomously  tracks  individual 
homogeneous  temperature  regions  and  intelligently  derives  displacement 
vectors  for  the  leading  edges  of  cloud-objects  from  GOES  IR  imagery.  CIRRUS 
I  processes  use  AI  methods  and  rules--but  it  is  not  an  expert  system.  Essential 
to  this  processing  is  the  transformation  of  the  digital  image  into  a  "symbolic" 
representation  defined  as  a  width-encoded  medial  axis.  The  symbolic  image 
representation  scheme  makes  it  possible  to  apply  image  understanding  by 
perceptual  grouping  concepts.  The  output  from  CIRRUS  I  is  called  the  Cloud- 
Tracked  Forward-Displacement  Vector  File  (CT-FD-VF).  This  vector  file  could 
be  used  as  input  to  a  vorticity  modeling  program  that  would  produce  the 
synoptic  wind  or  stream  flow  fields  over  North  America.  CIRRUS  I  lays  the 
foundation  for  automating  many  tedious  visual  analysis  tasks  that  are 
performed  by  a  human  where  shape  and  relative  proximity  information  are 
important  to  object  recognition.  For  example,  CIRRUS  I  could  be  extended  to 
recognize  weather  related  features  such  as  fronts,  troughs,  ridges,  areas  of 
high  and  low  pressure,  and  to  speculate  about  the  future  state  and  location  of 
these  features. 

1.  CIRRUS  I  DATA  FLOW  OVERVIEW 

A  data  flow  overview  for  CIRRUS  I,  shown  in  fig.  1,  depicts  a  satellite  receiver  collecting 
multi-temporal  GOES  IR  images  at  one  hour  intervals.  These  digital  images  (DT+1  and 
DT+2)  are  cut,  filtered,  and  sliced  into  eight  bins  ry  images-or  temperature  levels  of  which 
only  five  are  actually  processed.  We  refer  to  the  grayscale  shapes  as  cloud-regions.  Each  of 
the  five  levels  is  differenced  to  produce  a  third  set  of  images  (DT1-0).  Then,  each  image  set  is 
transformed  into  "symbolic"  image  sets,  referred  to  as  ST+1,  ST+0,  and  ST1-0.  A  symbolic- 
cloud-region  corresponds  to  a  width-encoded  medial  axis  (WEMA)  of  segmented  regions 
in  an  image.  The  WEMA  is  composed  of  data  that  is  organized  into  LISP  lists  for  the 
individual  cloud-regions.  Once  these  GOES  images  are  in  this  particular  type  of  symbolic 
representation,  the  symbolic-cloud-regons  in  ST+1  and  ST+0  are  initialized  as  cloud-oojects. 

Once  initialization  is  completed,  intelligent  cloud-object  tracking  is  performed.  This  process 
treats  all  the  cloud-objects  in  the  ST+0  image  set  as  target-clouds  which  must  be  found  or 
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FIGURE  1.  CIRRUS  I  overview. 

matched  in  the  ST+1  image  set.  The  cloud-objects  in  the  ST+1  image  set  are  potential 
candidate-clouds  that  may  or  may  not  correspond  to  one  or  more  target-clouds.  The 
tracking  process  first  identifies  a  finite  set  of  candidate-clouds  for  each  target-cloud.  Then  the 
tracking  process  proceeds  to  reduce  the  number  of  candidate-clouds  until  it  selects  one  or  more 
matched-clouds  for  each  target-cloud,  thus  completing  the  tracking  process.  A  target-cloud 
that  has  been  tracked  to  one  or  more  matched-clouds  is  now  referred  to  as  a  tracked-cloud. 

Once  the  tracking  process  is  completed,  cloud-region  displacement  vectors  are  computed  from 
the  differenced  image  set  ST1-0.  The  difference  calculation  is  performed  on  binary  images 
DT+1  and  DT+0  to  produce  an  image  array  with  values  +1, 0,  and  -1.  The  +1  represents  the 
non-overlapping  leading  edges  of  clouds-regions.  The  0  represents  the  overlapping  areas.  The 
-1  represents  the  non-overlapping  trailing  edges  of  cloud-regions.  Only  the  leading  edge 
regions  are  transformed  into  the  ST1-0  image  and,  subsequently,  used  to  calculate  cloud- 
region  displacement  vectors.  Since  these  vectors  only  indicate  magnitude  and  orientation, 
they  are  referred  to  as  orientation-vectors. 

Finally,  a  reconciliation  procedure  uses  knowledge  about  the  tracked-clouds  and  matched- 
clouds  to  determine  the  sign  or  direction  of  the  orientation-vectors.  The  result  is  the  Cloud- 
Tracked  Foward-Displacement  Vector  File  (CT-FD-VF),  thus  completing  the  CIRRUS  I 
processing. 

Once  the  CT-FD-VF  is  composed,  it  could  be  used  as  input  to  a  vorticity  modeling  program 
that  computes  the  synoptic  wind  or  stream  flow  fields  for  North  America.  This  stream  flow 
data  could  then  be  accessed  on-demand  by  the  Staff  Weather  Officer,  a  weather  forecast 
model,  or  any  Tactical  Decision  Aid  (TDA). 
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2.  DIGITAL  IMAGE  PRE-PROCESSING  (DIPP) 


The  digital  image  pre-processing  steps  are  shown  in  more  detail  in  fig.  2.  The  GOES  IR  image 
is  640  x  480  pixels  with  256  gray  levels.  The  image  is  cut  to  a  256  x  256  pixel  window.  The 
smaller  window  retains  most  of  North  America  while  eliminating  portions  of  the  image  where 
curvature  of  the  earth  is  great.  The  image  is  then  processed  by  a  5  x  5  median  filter  to  reduc  i 
noise.  The  image  is  "thick-sliced"  into  eight  equal  gray  levels  (i.e.,  a  32  gray  scale  range). 


GOES  IR 


640x460 


Cut  Image 

(256  x  256) 

v 

Apply  5x5 

Medium  Noise  Filter 

"  ’  \ 

Threshold  Image  Into 

8  Grayscale  Levels 
(0-7) 

v 

FIGURE  2.  Digital  image  pre-processing. 

Since  there  appears  to  be  little  information  in  the  warmest  gray  level  (i.e.,  range  0-31)  and  the 
two  coldest  levels  (i.e.,  range  192-256),  these  three  levels  are  discarded.  Five  gray  levels  are 
left  that  range  in  temperature  between  +28°C  and  -48°C.  Table  1  shows  each  gray  level  and 
its  associated  temperature  range.  (Parke,  1998). 


TABLE  1 

TEMPERATURE  RANGES  ASSOCIATED 
WITH  THE  FIVE  LEVELS  BEING  PROCESSED 


Gray  scale 

Level 

Sliced 

Range 

Temperature 
Range  (°C) 

1 

j. 

on  ca 

UU  “  u*x 

.  no 

vu  r  j.\j 

2 

65-  96 

+16  to  +  8 

3 

97  - 128 

+  8  to  -12 

4 

129  -  160 

-12  to  -24 

5 

161  -  192 

-24  to  -48 
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Once  the  first  and  second  Digital  image  at  Time  plus  ZERO  (DT+0)  and  ONE  (DT+1)  are 
processed  to  this  point,  all  five  levels  in  each  image  are  subtracted  to  produce  a  differenced 
image  set  (DT1-0)  for  each  of  the  five  levels.  The  difference  image  set  contains  only  regions 
that  do  not  overlap.  These  are  the  leading  edges  of  cloud-objects  that  are  moving.  When 
completed,  these  three  sets  of  digital  images  are  ready  to  be  transformed  into  a  symbolic 
representation. 

3.  DIGITAL  TO  SYMBOLIC  IMAGE  TRANSFORMATION  AND 
INITIALIZATION  OF  CLOUD-OBJECTS 

Illustrated  in  fig.  3  is  the  transformation  of  all  three  sets  of  digital  images  into  symbolic-cloud- 
regions  and  the  initialization  of  two  image  sets  (ST+1  and  ST+0)  as  cloud-objects. 

All  gray  levels  from  each  digital  image  set  (i.e.,  DT+1,  DT+2,  and  DT1-0)  are  passed  through  a 
FORTRAN  algorithm  called  the  Digital-to-Symbolic-Transformation  Algorithm  (DSTA)  to 
transform  each  level  from  a  digital  image  into  three  Symbolic  image  sets  (ST+1,  ST+2,  and 
ST1-0).  These  symbolic  image  sets  are  composed  of  numerical  data  in  a  LISP  list  format  (i.e., 
they  are  bounded  by  parenthesis).  The  data  defines  the  width-encoded  medial  axis  for  each 
cloud-region  (i.e.,  homogeneous  temperature  region)  at  each  of  the  five  levels. 

Each  individual  symbolic-cloud-region  in  each  of  the  five  levels  in  image  sets  ST+1  and  ST+2 
are  then  "initialized"  in  a  LISP  environment  as  a  cloud-object  with  attribute  slots  and  default 
information.  After  initialization,  each  thick-sliced  level  in  the  first  image  set  is  paired  with  its 
corresponding  level  in  the  second  image  set.  The  cloud-objects  in  both  levels  are  then 
processed  through  a  tracking  process,  an  attribute  calculation  process,  and  a  reconciliation 
process  to  ultimately  produce  the  Cloud-Tracked  Forward-Displacement  Vector  File 
(CT-FD-VF). 
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FIGURE  3.  Image  transformation  and  initialization. 
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4.  CLOUD-TRACKED  FORWARD-DISPLACEMENT  VECTOR  FILE 

(CT-FD-VF) 

Shown  in  fig.  4  are  the  remaining  processing  steps  needed  to  produce  the  CT-FD-VF.  In  the 
following  sections,  each  of  these  steps  is  described  in  detail. 
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FIGURE  4.  Intelligent  cloud  tracking  and  displacement  vector  reconciliation. 

4.1.  CLOUD-OBJECT  ATTRIBUTE  CALCULATIONS 

This  algorithm  identifies  all  possible  candidate-clouds  in  image  set  ST+1  that  are  in  close 
proximity  to  a  target-cloud  in  image  set  ST+0.  Close  proximity  is  determined  in  the  following 
manner.  During  initialization,  a  framing  box  is  created  from  the  WEMA  that  encloses  the 
target-cloud  within  some  tolerance  to  allow  for  maximum  displacement.  Similarly,  another 
framing  box  is  created  that  just  encloses  each  candidate-cloud.  Then,  all  the  cloud-objects  in 
the  ST+1  imr  Te  that  overlap  the  target-cloud  box  are  candidate-clouds. 

The  size  and  length  of  candidate-clouds  are  computed  using  the  WEMA.  The  size  of  a  cloud- 
object  is  defined  as  a  diagonal  between  the  upper-left-most  corner  and  the  lower-right-most 
comer  of  the  frame  box  that  encloses  the  WEMA  skeleton.  The  length  of  a  cloud-object  is 
defined  as  the  summation  of  all  arcs  in  the  WEMA. 
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The  gross  overlapping  area  between  the  remaining  candidate-clouds  and  the  target-cloud  is 
the  difference  between  the  box  that  encloses  the  target-cloud  and  the  box  that  encloses  one  of 
its  candidate-clouds.  The  area  of  overlap  is  normalized  by  t>  .  area  of  the  target-cloud  box  and 
then  the  area  of  the  candidate-cloud  box,  thus  producing  two  measures  of  the  gross  overlap  for 
each  candidate-cloud.  These  measures  are  then  attached  to  the  cloud-objects  for  use  during 
the  behavior  classification  process. 

4.2  INTELLIGENT  CLOUD-OBJECT  BEHAVIOR  CLASSIFICATION 

The  next  processing  step  is  to  determine  the  behavior  of  the  cloud-objects.  *  The  objective  here 
is  to  classify  how  a  target-cloud  might  be  evolving  so  that  it  can  be  used  to  select  by  logical 
inference  the  correct  candidate-cloud(s)  when  possible,  and  to  logically  infer  the  most  relevant 
cloud-candidate(s)  when  necessary. 

Cloud  behavior  is  determined  by  using  ancillary  information  and  a  dozen  or  so  rules  to  classify 
the  target-clouds  into  one  of  five  categories  (i.e.,  Sub-Cloud,  Super-Cloud,  Hyper-Cloud,  Meta- 
Cloud,  and  Null).  The  distinction  between  behavior  classes  is  illustrated  in  figs.  5-9.  The 
region  labeled  "A"  is  the  target-cloud  and  the  region  labeled  "D"  is  the  correctly  tracked  cloud- 
candidate  (i.e.,  the  tracked-cloud). 

Cloud  Behavior-1  (CB-1)  is  illustrated  in  fig.  5  where  all  of  D  is  a  subset  of  A.  CB-1  is  the 
situation  where  target  cloud  A  is  breaking  up  into  one  or  more  subclouds  (i.e.,  one-to-one  or 
one-to-many).  In  other  words,  D  is  a  Sub-Cloud  of  A  CB-2  is  illustrated  in  fig.  6  where  D  is  a 
superset  of  all  of  A  CB-2  is  the  situation  where  candidate  cloud  D  has  been  formed  by  the 
merging  of  more  than  one  cloud  (i.e.,  many-to-one).  In  other  words,  D  is  a  Super-Cloud  of  A 
CB-3  is  illustrated  in  fig.  7  where  part  of  D  is  a  subset  of  A  CB-3  is  the  situation  where 
candidate  cloud  D  has  been  formed  by  the  breaking  up  of  target  cloud  A  and  the  merging  with 
one  or  more  other  clouds  (i.e.,  few-to-many)  to  form  cloud  D.  In  other  words,  D  is  a  Hyper- 
Cloud  of  A  CB-4  is  illustrated  in  fig.  8  where  D  is  a  superset  of  part  of  A  CB-4  is  the 
situation  where  candidate  cloud  D  has  been  formed  by  the  merging  of  part  of  cloud  A  with 
other  clouds  (i.e.,  many-to-few).  In  other  words,  D  is  a  Meta-Cloud  of  A.  CB-5  is  the  Null 
category  to  account  for  those  candidates  that  do  not  have  any  relevant  behavior. 

When  the  behavior  of  the  target-clouds  is  finally  determined,  it  is  attached  to  the  target-clouds 
in  the  ST+0  image  set.  The  behavioral  knowledge  can  then  be  used  to  logically  infer  which 
candidate-cloud(s)  are  the  matched-cloud(s).  When  the  target-cloud  is  matched  to  its 
candidates  (i.e.,  matched-clouds),  the  target-cloud  is  referred  to  as  the  tracked-cloud. 

4.3  CLOUD-OBJECT  DISPLACEMENT  VECTOR  FILE 

The  symbolic  differenced  image  set  ST1-0  is  used  to  compute  the  orientation-vectors  for 
leading  edges  of  the  cloud-regions  in  ST+1  and  ST+0.  Typically,  the  WEMA  for  a  leading  edge 
is  distinctly  elongated.  In  this  case,  the  cloud-region  orientation-vector  is  perpendicular  to 
each  of  the  major  WEMA  arcs.  However,  no  attempt  is  made  to  determine  which  direction  the 
orientation-vectors  are  actually  pointing  (i.e.,  the  sign).  This  problem  is  resolved  by  the 
reconciliation  process  described  in  the  next  section. 


^Examples  of  typical  cloud  behavior  are  as  follows:  growing,  merging,  converging, 
diverging,  shrinking,  dissipating,  and  breaking-up  without  dissipating.  In  CIRRUS  I  these 
behaviors  are  grouped  into  five  categories  as  follows:  breaking-up,  merging,  breaking-up-and- 
merging,  and  merging-and-breaking-up,  and  no  discemable  behavior  (i.e.,  null). 
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BREAKING  UP  (ONE  TO  MANY)  OR  (ONE  TO  ONE) 


MERGING  (MANY  TO  ONE) 


FIGURE  5.  Cloud  behavior  1:  sub-cloud  FIGURE  6.  Cloud  behavior  2:  super-cloud 
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4.4  KNOWLEDGE  RECONCILIATION 

By  the  time  the  reconciliation  process  is  ready  to  begin,  a  considerable  amount  of  information 
or  knowledge  is  readily  available.  For  example,  the  locations  of  the  orientation-vectors  for  all 
the  leading-edges  of  the  overlapping  cloud-regions  are  known.  The  tracked-clouds  and  their 
locations  are  known.  The  matched-clouds  and  their  locations  are  known.  The  behavior  of  all 
the  target-clouds  is  known.  This  knowledge  is  now  reconciled  to  determine  the  direction  that 
the  orientation-vectors  are  pointed. 

Each  orientation-vector  is  extended  in  both  directions  until  it  crosses  the  major  medial  axis 
(i.e.,  a  major  WEMA  arc)  of  a  matched-cloud  and  its  corresponding  tracked-cloud.  The 
direction  of  the  orientation-vector  is  from  the  tracked-cloud  toward  the  matched-cloud.  The 
process  is  repeated  for  all  orientation-vectors.  Thus,  the  output  from  this  last  processing  step 
is  the  Cloud-Tracked  Forward-Displacement  Vector  File  (CT-FD-VF)  which  consists  of  all  the 
cloud  displacement  vectors  for  the  leading  edges  of  all  tracked-clouds  for  all  five  gray  levels. 
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5.  CONCLUSION 


It  is  important  to  note  that  the  goal  of  CIRRUS  I  is  to  extract  systematic  wind  flow 
information  from  the  movement  (i.e.,  the  displacement)  of  clouds  as  viewed  from  satellite 
imagery.  Obviously,  as  little  unsystematic  error  as  possible  is  desired.  However,  if  the  CT- 
FD-VF  is  directly  compared  to  a  synoptic  stream  flow  map,  one  could  reasonably  expect 
correlation,  systematic  bias,  and  non-systematic  error.  Although  we  once  considered 
determining  the  "accuracy1'  of  the  CT-FD-VF  by  comparing  it  to  a  synoptic  stream  flow  map, 
we  finally  realized  that  it  was  actually  an  irrelevant  comparison  given  the  intended  purpose  of 
CIRRUS  I. 

CIRRUS  I  is  supposed  to  provide  yet  another  source  of  candidate  predictor  variables  that 
would  be  processed  by  the  GO  PAD  forecast  model  development  system.  Consequently, 
CIRRUS  1  has  been  biased,  relative  to  a  synoptic  wind  flow  map,  because  it  produces 
displacement  vectors  only  for  the  leading  edges  of  cloud  regions.  While  it  may  be  very 
desirable  for  forecasting,  this  bias  may  appear  to  be  unsystematic  error  if  these  CIRRUS 
vectors  were  compared  to  a  synoptic  steam  flow  map.  Therefore,  an  accuracy  measurement  for 
CIRRUS  I  was  not  performed.  It  is  speculated  that  a  neural-net-based  weather  forecasting 
model  like  GOPAD  will  remove,  to  a  great  extent,  the  systematic  bias,  and  mitigate  the 
unsystematic  error  to  some  degree,  even  if  CIRRUS  I  and  the  vorticity  modeling  program 
produce  wind  flow  vectors  that  appear  to  have  error  relative  to  a  stream  flow  map. 

The  ultimate  test  of  the  usefulness  of  CIRRUS  I  for  mesoscale  forecasting  is  the  quality  of 
systematic  information  contained  in  its  candidate  predictor  variables  that  can  contribute  to 
the  forecast  relative  to  the  information  available  from  other  sources.  Therefore,  the  best  way 
to  correctly  determine  the  value  of  the  CIRRUS  I  and  the  vorticity  modeling  program  is  to 
produce  a  multi-year  historical  synoptic  wind  flow  data  base  and  measure  the  statistical 
contribution  of  CIRRUS  I  as  a  source  of  predictor  variables  relative  to  the  candidate  predictor 
variables  from  other  sources  (e.g.,  rawinsonde,  mesonet,  NGM,  barotropic,  etc.).  The  GOPAD 
model  development  system  is  an  ideal  tool  for  such  a  task  (Young  and  Lampru,  1989). 
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CIRRUS -I  PERFORMANCE 


1.0  INTRODUCTION 

The  performance  of  CIRRUS-I  is  shown  in  the  following  series  of  photographs. 1  These 
photographs  illustrate  the  results  obtained  from  the  following  image  processing 
procedures: 

(a)  digital  image  pre-processing 

(b)  digital  image  to  symbolic  image  transformation 

(c)  symbolic  cloud-object  behavioral  classification 

(d)  symbolic  cloud-object  structural  matching 

(e)  calculation  of  leading  edge  orientation  and  magnitude  displacements 

However,  before  reading  this  Appendix,  it  may  be  helpful  to  refer  to  Appendix  D  first  for 
a  complete  description  on  how  CIRRUS-I  operates. 

2.0  DIGITAL  IMAGE  PRE-PROCESSING 


^ These  photographs  were  taken  with  a  Minolta  X  700,  manual  focus  camera  with  F8, 1  /8  second  exposure  on  Kodak  Gold 
400/32  aim. 
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This  photograph  shows  a  raw,  ZA-unerihanced,  IR,  640x400,  GOES  Central  image  that 
was  captured  on  June  22,  1989  at  20:01  hours  GMT,  This  image  is  referred  to  as  T+0. 
The  dark  yellow  color  represents  warm  clouds  and/or  thin  clouds  in  which  me  warm 
temperature  from  the  earth's  surface  have  bleed-through.  The  yellow-green  and 
greenish-blue  colors  represent  inteimediate  temperatures,  while  the  darker  blue 
represents  the  coldest  clouds. 

2.2  CUTTING  AND  F  ILTERING 


The  raw  GOES  IR  image  is  cut  with  the  following  offsets:  x=256  and  y=32.  This  cutting 
produced  a  sub-image,  shown  on  the  left,  that  is  256x256  pixels  in  size.  After  cutting, 
a  5x5  median  filter  is  applied  to  produce  the  image  shown  on  the  right  in  which  the  IR 
regions  are  more  homogeneous. 
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2.3  LEVEL  SLICING 


The  T+0  Image  and  the  T+l  are  Level  1  slices  taken  after  the  median  filter  is  applied. 
The  temperature  range  for  this  level  Is  +16°  to  +28°C.  Notice  the  changes  In  shape  and 
the  movement  of  the  cloud  regions  from  one  slice  to  the  next. 
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2.4  DIFFERENCING  LEVEL  PAIRS 


The  sequential  slices  are  now  differenced  to  identify  the  leading  edges  of  clouds  (+1),  the 
trailing  edges  of  clouds  (-1),  and  the  overlapping  regions  (0).  This  photograph  shows  the 
leading  edges. 


3.0  DIGITAL  TO  SYMBOLIC  IMAGE  TRANSFORMATION 

3. 1  TRANSFORMATION  OF  LEADING  EDGES 


When  differencing  is  completed,  the  differenced  image  and  all  binary  slices  are 
transformed  into  a  symbolic  or  WEMA  representation.  This  photograph  displays  the 
medial  axis  of  the  leading  edges  redrawn  from  the  WEMA.  The  width  encodatlon  is  not 
displayed. 
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3.2  CALCULATION  OF  LEADING  EDGE  ORIENTATION  AND  MAGNITUDE 

DISPLACEMENT 


Trailing  Edge  Overlap  Leading  Edge 


Cloud  TO  Cloud  T1 


This  diagram  shows  how  the  WEMA  for  the  leading  edge  of  a  cloud  are  used  to  calculate 
orientation  and  magnitude  displacement  data.  The  orientation  is  calculated  from  the 
perpendicular  bisection  of  the  WEMA  The  magnitude  is  calculated  from  the  width  data 
explicit  in  this  symbolic  representation.  During  intelligent  reconciliation,  the  direction 
of  movement  is  derived  by  using  behavioral  and  structural  information  to  determine  the 
track  of  a  target  cloud  to  all  its  candidate  clouds. 
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3.3  CALCULATION  OF  CLOUD  DISPLACEMENT  DATA 


This  photograph  shows  the  leading  edge  displai  cmcnl  data  drawn  to  scale. 
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4.0  IMAGE  SET  1 


The  overall  goal  CIRRUS-I  Is  (a)  to  classify  the  behavior  of  cloud-objects,  (b)  to 
Saucturalfy  match  target-cloud  arcs  to  candidate-cloud  arcs,  (c)  to  intelligently  use  all 
the  accumulated  behavioral  and  structural  knowledge  to  produce  the  Cloud-Tracked 
Forward-Displacement  Vector  File  (CT-FD-VF).  Behavioral  classification  determines 
how  a  target-cloud  might  be  evolving  so  that  this  information  can  be  used  to  decide 
which  candidates  will  be  structurally  matched.  Structural  matching  determines  which 
arcs  in  the  target’s  WEMA  correspond  to  arcs  in  a  candidate's  WEMA.  This  information 
is  important  for  two  reasons.  First,  structural  matching  helps  the  behavioral 
classification  process  to  correctly  discriminate  between  relevant  and  irrelevant  clouds. 
Second,  structural  information  is  used,  along  with  behavioral  information,  during  an 
intelligent  reconciliation  process  to  determine  the  direction  in  which  each  target  cloud  is 
displacing. 

In  the  series  of  photographs  that  follow,  pairs  of  symbolic  images  are  shown  which  were 
taken  at  one-hour  intervals.  In  each  photograph,  the  left  image  contains  the  medial 
axis  of  the  target  clouds,  while  the  right  image  contains  the  medial  axis  of  the  candidate 
clouds. 


4. 1  OVERLAY  OF  TARGET  AND  CANDIDATE  CLOUDS 

In  the  following  photograph,  the  medial  axes  from  the  T+0  slice  and  the  T+l  slice  at 
Level  1  are  overlaid  so  spatial  and  shape  information  can  be  observed.  The  target 
clouds  are  shown  in  blue;  the  candidate  clouds  are  shown  in  green;  and  the  overlapping 
axes  are  shown  in  aqua. 
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4.2  BEHAVIORAL  CLASSIFICATION  (SUB-CLOUD  AND  HYPER-CLOUD) 


On  the  left,  a  massive  target  cloud,  shown  in  white.  Is  selected  to  begin  the  tracking 
process.  Note  that  cloud-objects  are  defined  as  a  single,  unbroken  linkage  of  arcs. 

On  the  right,  the  T+l  is  shown  with  the  color-coded  results  obtained  after  CIRRUS-I  has 
completed  behavioral  classification.  The  blue  arcs  are  non- candidate  clouds.  The  two 
red  structures  are  candidate  clouds  that  broke  off  from  the  target  cloud.  They  are 
classified  as  sub-clouds  (i.e.,  dissolving  and  breaking  up).  The  magenta  cloud 
structure  apparently  has  broken  off  from  the  target  cloud  and  combined  with  another 
cloud  that  was  north  of  the  target  cloud.  It  is  classified  as  a  hyper-cloud  (i.e.,  breaking 
up  and  merging).  Although  it  is  difficult  to  see,  there  are  scattered  brown  cloud 
structures  which  are  located  close  to  the  magenta  cloud  and  that  were  considered  to  be 
candidates.  After  classification  was  completed,  these  clouds  were  determined  to  be 
irrelevant. 
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4.3  STRUCTURAL  MATCHING  (SUB-CLOUDS  AND  HYPER-CLOUDS) 


When  behavioral  classification  is  completed,  a  WEMA  structural  match  between  the 
target  cloud  and  the  behaviorally  classified  candidates  is  performed.  The  target  cloud'c 
arcs  that  structurally  match  the  arcs  in  the  candidate  clouds  are  shown  in  red  and 
pink  in  both  images.  The  intermixed  white  fragments  in  the  target  cloud  now  depict 
the  arcs  where  no  structural  match  is  found.  Notice  the  long,  horizontal,  white  arc  in 
the  lower  portion  of  the  target  cloud.  This  arc  cannot  be  matched  by  a  similar  arc  in 
any  candidate  cloud  because  that  portion  of  the  target  cloud  apparently  dissolved. 
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4.4  BEHAVIORAL  CLASSIFICATION  (SUB-CLOUD  AND  META-CLOUD) 


The  white  target  cloud  Is  behaviorally  matched  to  three  candidate  clouds  shown  on  the 
right  in  green,  red,  and  brown. 

The  green  candidate  cloud  is  classified  as  a  meta-cloud  (i.e.,  merging  and  breaking  up) 
because  it  has  combined  with  a  cioud  to  the  south  which  is  breaking  off  another,  larger 
structure.  The  red  candidate  cloud  is  classified  as  a  sub-cloud  (i.e.,  breaking  up) 
because  it  broke  off  the  northern  part  of  the  target  cloud.  The  brown  candidate  cloud 
which  Is  very  difficult  to  see  Is  irrelevant  because  it  is  too  far  away  from  the  target  cloud 
to  have  broken  off. 
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4.5  STRUCTURAL  MATCHING  (SUB-CLOUD  AND  META-CLOUD) 


The  target  cloud  Is  now  colored  green  to  indicate  the  arcs  that  are  structurally  matched 
to  the  green  arcs  in  the  candidate  cloud.  Likewise,  the  red  arcs  in  the  target  cloud  are 
structurally  matched  to  the  red  arcs  that  broke  off  to  the  north. 
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4.6  BEHAVIORAL  CLASSIFICATION  (SUPER-CLOUD) 


The  small,  white  target  cloud  on  the  left  Is  behaviorally  matched  to  two  candidate 
clouds  shown  In  yellow  and  brown  on  the  right.  The  large,  yellow  candidate  Is 
classified  as  a  super-cloud  (i.e.,  merging  and/or  growing)  because  It  combines  with 
several  cloud  regions  that  were  north  of  the  target  cloud.  A  small,  brown  candidate 
cloud,  which  is  very  difficult  to  see,  was  determined  to  have  no  relevant  behavior. 
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The  white  target  cloud  Is  behaviorally  matched  to  two  candidate  clouds  shown  in  yellow 
and  brown.  The  yellow  cloud  is  classified  as  a  super-clori  (l.e.,  merging  and/or 
growing)  because  it  has  a  larger  are„  than  the  target  cloud  suggesting  that  the  ‘^.rget 
cloud  has  grown.  A  small,  brown  candidate  cloud  to  the  northwest  of  the  yellow 
candidate  cloud  is  determined  to  be  Irrelevant. 

4.9  STRUCTURAL  MATCHING  (SUPER-CLOUD) 


The  yellow  target  cloud  arcs  are  structurally  matched  to  the  yellow  candidate  cloud 
arcs.  Notice  that  the  white  arc  in  the  upper  middle  portion  of  the  target  cloud  is  not 
matched,  while  the  arc  at  the  tip  is  matched.  The  while  arc  is  considered  to  be 
unmatched  because  its  structure  Is  too  different  fr^.n  the  candidate  cloud.  However, 
the  tip  of  the  arc  was  matched  because  it  is  similar. 
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4. 10  BEHAVIORAL  CLASSIFICATION  (SUB-CIXDUD  AND  HYPER-CLOUD) 


The  white  target  cloud  Is  behaviorally  matched  to  eight  candidate  clouds  shown  in  red, 
magenta,  and  brown.  The  red  clouds  are  classified  as  sub-clouds  because  they  broke 
up  into  smaller  clouds.  The  magenta  cloud  is  correctly  classified  as  a  hyper-cloud,  yet 
Incorrectly  selected  as  a  relevant  candidate.  Evidently,  CIRRUS-I  considered  the 
magenta  cloud  to  be  a  candidate  which  might  have  broken  off  from  the  target  cloud  and 
merged  with  other  clouds.  However,  the  magenta  cloud  Is  actually  formed  by  the 
combination  of  the  blue  clouds  to  the  north  of  the  target  cloud.  As  will  be  seen  in  the 
next  photograph,  this  temporary  mismatch  is  corrected  during  the  structural  matching 
process. 
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4. 1 1  STRUCTURAL  MATCHING  (SUB-CLOUD  AND  HYPER-CLOUD) 


The  red  arcs  In  both  Images  show  the  arcs  that  structurally  match.  Notice  that  the 
magenta  hyper-cloud  Is  determined  to  be  Irrelevant  (unmatched)  to  the  target  cloud 
and  Is  now  correctly  colored  blue.  The  white  arc  on  the  tip  of  the  target  cloud  Indicates 
it  Is  not  structurally  matched  to  the  red  candidate  cloud  because  [we  assume)  its 
displacement  exceeds  the  maximum  allowed. 
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5.0  IMAGE  SET  2 


The  remaining  GOES  IR  images  were  taken  on  January  23,  1989,  at  11.00  and  12:00 
hours  GMT.  The  first  photo  shows  the  results  obtained  by  behavior  classification  while 
the  second  shows  structural  matching.  This  series  cf  photographs  provides  addit/  nal 
tracking  scenarios  to  demonstrate  the  robustness  of  CIRRUS-I. 


5. 1  OVERLAY  OF  TARGET  AND  CANDIDATE  CLOUDS 
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5.2  BEHAVIORAL  CLASSIFICATION  (SUB-CLOUD  AND  HYPER-CLOUD) 


BEHAVIORAL  CLASSIFICATION  (META-CLOUD) 


STRUCTURAL  MATCHING  (META-CLOUD) 


BEHAVIORAL  CLASSIFICATION  (SUB-CLOUD  AND  HYPER-CLOUD) 


STRUCTURAL  MATCHING  (SUB-CLOUD  AND  HYPER-CLOUD) 


